diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index dbcd7ca01..895c41545 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -30,7 +30,10 @@ jobs: - run: npx playwright install --with-deps chromium # The extensions as the portal ships them, built: the tests list the ones in extensions/ by the halves they have. - run: npm run build:extensions + # OFFICE_IT_DOCKER=1 turns on 3 docker-gated office integration tests; they run in-process against a synthetic DS stand-in and only need a reachable docker CLI, which runners have. - run: npm test + env: + OFFICE_IT_DOCKER: "1" # The type check of the web app and its build, which is also what the image runs, the extensions' builds, and # the line between them and core (scripts/check-boundaries.mjs). diff --git a/PLAN-office-280.md b/PLAN-office-280.md new file mode 100644 index 000000000..4170ae5dd --- /dev/null +++ b/PLAN-office-280.md @@ -0,0 +1,674 @@ +# Implementation plan: pithagoras-office (issue #280) + +Status: proposal -- revised 2026-10-23 three times: (1) technical review +fixes, (2) own document library + live co-editing, (3) agent/project ownership +with @mention grants + global Office tab (user-confirmed decisions). +Ready for re-review. +Issue: https://github.com/thecodacus/pithagoras/issues/280 — "Concept: pithagoras-office extension — embedded OnlyOffice Document Server (sidebar + agent tools + comment wake)" + +## 1. Goal + +Add an opt-in **pithagoras-office** extension that embeds OnlyOffice Document +Server (DS) in a portal-managed container and gives the portal three things: + +1. A sidebar page where users open real Office files (.docx/.xlsx first, .pptx + later) in the full DS editor. +2. Agent tools so agents can read and edit those same files (OOXML-aware), + gated by ownership + mention-grants and revisions; an open human session + triggers forcesave-first ordering, not refusal (co-editing is the default, + §5.7). +3. Wake-on-mention: a comment containing @agent / @ / + @project: wakes the named agent or project in its home chat with the + quoted comment and grants it access to that one document (docs ping + pattern). +4. A global **Office tab** on the main nav -- the full library, styled like an + office suite (not portal card chrome), with a full-bleed editor. + +Motivation (from issue): pithagoras-docs is deliberately barebones markdown; +office documents are a real user need where "good enough" means a real Office +editor. DS was chosen over Collabora because of the callback/locking model and +the official ecosystem (see §2 — the MCP assumption turns out to be weaker +than the issue states). + +## 2. Decisions on the issue's open questions + +### 2.1 Storage layout → own document library in the extension's dataDir (NOT chat folders) + +Documents live in **pithagoras-office's own folder** -- `portal.dataDir()` +(verified API, packages/extension-api/server.d.ts:844; precedent: understory +keeps its settings there): `docs//.<ext>` plus a metadata index +(`index.json`; per-doc JSON, SQLite only if it ever needs queries). docId is +generated (slug + short random suffix) -- never a chat-relative path. Chat +folders are untouched: office works like real office software with its own +library; documents reach agents and projects through ownership and mentions, +not through chat membership. + +**Ownership / grants (user-confirmed model)**: the discriminator is folder-based -- `portal.folders.list()` returns every agent home and project as `AgentFolder{path,name,kind:"agent"|"project"|"chat",home?}` (server.d.ts:91; a `Chat` carries no kind), and the chats.extend factory resolves each chat's workspace to that folder kind. Each index +entry carries `owner: <home chat id> | null` -- the home of the agent/project +that created the doc (docs a human creates from the global tab are ownerless) +-- plus `grants: [{chatId, commentId, at}]`. Rules: +- An agent/project may edit only docs it **owns** or was **granted**; its + office_* tools and home-chat panel see exactly that set. +- Ownership is standing access for the creating agent/project's context; any + other agent/project gets access to **that one file only** by being mentioned + in a comment of it (§5.4) -- never library-wide. +- Human UI access (global tab, opening documents, import/export) is + unrestricted; only agent tooling is gated. + +Import/export to chat folders are first-class operations (copy the file in or +out through portal.files), so nothing is stranded and Files-extension users +got a copy whenever they want one there. + +SUPERSEDED by PR #307 (the implemented extension): manual import is GONE. +The library instead shows every .docx/.xlsx found across the agent homes and +projects -- `portal().folders.list()` is the root source -- indexed automatically +at boot (after settleKept) and on POST /documents/rescan, IN PLACE (DocMeta.path); +export to chat folders remains deferred as listed below. DS's own storage +(`/var/www/onlyoffice/Data` volume) stays an ephemeral working cache -- every +save round-trips to the library (§4 mechanism), so wiping the container loses +nothing. + +Rejected alternatives: (a) chat-folder storage (first draft of this plan) -- +coupled every document to one chat, made "restrict to a project" awkward, and +left no room for library semantics (rename history, trash, versions); +(b) DS's own storage (issue's option B) -- opaque, un-backupable, breaks the +stable filesystem path the agent tools need. +M1 verifies dataDir gets the same persistence/backup treatment chat folders +do; if not, a note lands in the docs. + +### 2.2 Agent surface → built-in OOXML tools, NOT MCP + +The issue assumed the official MCP server (ONLYOFFICE/docspace-mcp) could serve +the agent. **Verified against the repo (public, MIT): it does not.** Its README +shows it targets ONLYOFFICE Docspace — the cloud SaaS product — requiring +`DOCSPACE_BASE_URL` + `DOCSPACE_API_KEY`, and its toolset is workspace-level +(rooms, folders, permissions, people directory), not document-content editing. +Pointing it at our private embedded DS would be wrong twice: wrong product and +wrong granularity, plus data would have to leave the machine. + +So v1 registers plain extension tools in agent/project home chats only via +`portal.chats.extend((pi, chat) => …)` (the factory knows its chat; ownership +scoping in §5.5), OOXML-aware and docs-extension-style +(named by id, base-revision writes, coded error prefixes). This keeps edits +deterministic, local, and gate-able on lock state. + +Future (post-v1, still open): a community MCP for on-prem DS could be wired +with `portal.mcp.set("…", {command:"docker", args:[…]})` behind an opt-in +setting; Docspace-cloud mode if a user has their own key. Not in this plan. + +### 2.3 Scope → multi-document, single human identity per session + +- Multi-document: any number of documents in the library (§2.1); each open + editor is tracked independently by its DS `document.key` (key = docId + + revision hash). +- Co-editing inside the editor works as-is (DS handles it). Our lock model only + tracks **someone vs. nobody** editing a document (from callback `users` + lists), which is all the agent needs. +- v1 uses one portal user identity for the DS session (`editorConfig.user` + from the requesting portal session). Per-user identities/profiles: later. + +### 2.4 Packaging → opt-in package requiring pithagoras-docker + +`package.json`: name `pithagoras-office`, keywords `["pi-package"]`, +`"pi": {}`, `"pithagoras": { client, server, style }`, with the dependency +**nested** as `"pithagoras": { ..., "requires": { "pithagoras-docker": +"^1.1.0" } }` (core reads requires only under that key -- a top-level +`requires` would not resolve; precedent: understory's package.json). +**Not** added to `extensions/defaults.json` — it stays opt-in; enabling is an +explicit settings-page action that starts the container and pulls the image +(~GB-scale, ~4 GB RAM recommended at runtime — surfaced as a warning before +start). Extensions are standalone npm packages (not root workspaces); the root +`scripts/build-extensions.mjs` picks the new directory up automatically — +verify in M1. + +### 2.5 Decisions (confirmed on re-review + one picked default) + +- **D1 ownership (confirmed)**: documents are owned by the agent/project that + created them -- not bound to arbitrary chats; human-created docs are + ownerless. Model in §2.1/§5.4. +- **D2 co-editing (confirmed)**: live agent co-editing is ON by default and + must always work (§5.7, forcesave-first ordering); a setting to disable it + remains as an escape hatch. +- **D3 grant lifetime (picked, flag for re-review)**: a mention-grant (§5.4) + stays on the doc until revoked from the UI -- multi-turn tasks need it; + strict per-comment expiry deferred to M4. Grants are scoped to that one + file and every grant/revoke is audited. + +## 3. Architecture + + +--------------------- portal UI (browser half) ----------------------+ + | global nav page "Office": library browser <-> full-bleed editor | + | OfficeEditor: DocsAPI.DocEditor placeholder + minted config/JWT | + | per-home chat.panel (owner/granted docs), tool cards, settings page | + +-------------+ ^---------------+ + | session auth (/api) | DS origin (see + v | reachability, M0) + +----------------- portal server half ------------------------------+ + | A) /api/x/pithagoras-office/... -- session-gated (UI): | + | documents CRUD * state * editor-config (mints JWT) | + | B) standalone loopback listener (127.0.0.1, in-process, | + | OUTSIDE /api so the login middleware never sees it): | + | GET /doc/<ticket> <- DS container fetches document.url | + | POST /callback <- DS POSTs status/actions/users/url/changesurl | + | C) shared state: lock state machine * wake queue (docs ping pattern) + | * tools via chats.extend (lock- + revision-gated) | + +------+---------------------------+---------------------------------+ + | document bytes (HMAC ticket, no cookie needed) | + v | POST callback; GET the saved file + +-------+--------------------+------ docker: onlyoffice/documentserver:<pinned> + | editor UI served by DS stock nginx -- under shared networking it | + | listens :80 on ALL interfaces of the shared netns (PortBindings are | + | ignored); browser reach = local-only or portal proxy route, M0 #2 | + | internal storage volume (cache only) | + +---------------------------------------------------------------------+ + office library in the extension dataDir: docs/<docId>/ + index.json + -- source of truth (owned by the creating agent/project; mention grants + per file; NOT in chat folders) + +Components and where they copy from: + +| Component | Source pattern in repo | +|---|---| +| host.ts / container lifecycle | extensions/understory/src/server.ts + src/server/host.ts (setPortal/portal()/docker(); adopt on boot; setInterval(lookAfter, 30_000).unref(); dependents don't self-stop) | +| Docker API usage | extensions/docker/src/api.ts via portal.extensions.use<Docker>("pithagoras-docker"): images.ensure (progress), containers.create/start/stop/logs/exec/adopt/inspect, volumes.*, sharedNetwork() | +| Sidebar page + panel + tool cards | extensions/docs/web/client.js (nav.page documents -> DocumentsPage; chat.panel "documents"; one chat.tool entry per tool with "Open in Documents" button) | +| Wake on mention | extensions/docs/ping.js: server queues {text, details} per chat -> portal.chats.tell(chatId,{type:"ping"}) rings bus channel UI_RECEIVE="pithagoras:v1:ui:receive" -> per-chat pi extension drains queue with pi.sendMessage({customType, content, display:true, details}, {deliverAs:"followUp", triggerTurn:true}) | +| Tool style (ids not paths, base revision, coded errors) | extensions/docs/tools.js header + error codes (not_found/exists/revision_mismatch/too_large/invalid/refused -- office adds: unsupported_format, corrupt; `locked` is reserved, only reachable if co-editing is disabled or for M4 xlsx writes) | +| package skeleton, build.mjs (esbuild from repo node_modules; deps.js missingDependencies -> npm ci --ignore-scripts) | extensions/docs/ | + +## 4. Verified OnlyOffice integration facts (official docs, current state) + +Source: https://api.onlyoffice.com/docs/docs-api/usage-api/callback-handler +(fetched and read during exploration; current docs describe DS 10.x behavior). + +- **Editor embedding (current documented entry point)**: the page loads the + script /web-apps/apps/api/documents/api.js and calls + new DocsAPI.DocEditor(placeholder, config) -- DS replaces the placeholder + with its own iframe to /web-apps/apps/documenteditor/main/index.html. A raw + ?config=<JWT> query URL is NOT the documented embedding (older builds' path + was webapps/onlyoffice/editor/index.html; unverified on 10.x -- M0 may note + it, but v1 uses api.js + DocEditor, which is also required for client-side + events). Config object: top-level documentType ("word"|"cell"| + "slide"|"pdf"|"diagram"), type:"embedded", title, height/width, token field, + document{fileType,key,title,url}, editorConfig{callbackUrl,mode,user{id,name}, + customization...}, optional client-side events{}. Since DS 7.1 the signed + parameter list is "strictly regulated": sign exactly what you send (the + official sample includes document.permissions and editorConfig.mode -- verify + the exact set on the pinned build in M0). +- **Callback body** (POST to callbackUrl): required key + status; plus + actions[] ({type: 0 disconnect | 1 connect | 2 forcesave-click, userid}), + users[], url ("link to the edited document to be saved", present for status + 2/3/6/7), changesurl (changes archive, status 2/3/6/7), + history{changes,serverVersion} (status 2/3), filetype, forcesavetype + (status 6/7: 0 command service, 1 Save button, 2 timer, 3 form submit), + formsdataurl (status 6 + forcesavetype 3), userdata. +- **Status semantics** (current numbering -- differs from older blog posts): + - 1 = being edited; sent on every user connect/disconnect (our lock signal, with actions/users) + - 2 = ready for saving -- closed after changes (~10 s after close); host downloads url and persists + - 3 = document saving error + - 4 = closed with no changes (-> unlock) + - 6 = force-saving performed, editing continues (download + persist, stay locked) + - 7 = force-saving error +- **Save-back mechanism (verified in the official Node.js example)**: on status + 2 the storage service does fetch(req.body.url) -> streams to disk, then must + answer {"error":0}; anything else makes the editor show an error. There is no + other push of the full file -- this GET of url IS the save. +- **JWT**: since DS 7.2 JWT is enabled by default (auto-generated secret). + The internal local.json has per-direction enable/secret sections + (inbox/outbox/session), but the community Docker image's documented env vars + are only JWT_ENABLED / JWT_SECRET / JWT_HEADER / JWT_IN_BODY -- a single + JWT_SECRET covers config tokens AND callback verification in every direction + (no per-direction env vars; editing the container-local local.json is + regenerated on restart, so env is the documented mechanism). v1 therefore + generates one secret and passes it as JWT_SECRET; DS still verifies our + signed configs and signs/verifies callback traffic with it. A per-direction + two-secret setup (inbox for config signing, outbox for callback verification) + would need local.json mount/patching -- deferred unless M0 proves a supported + way on the pinned image. +- **Client-side DocEditor methods (documented, fetched)**: the host page's + DocsAPI.DocEditor object exposes -- refreshFile(newConfig): "updates the file + version without reloading the editor" (officially called in answer to the + editor UI's onRequestRefreshFile event; parameters must stay documentType/ + type/events); refreshHistory({currentVersion, history:[{created,key,user, + version, changes?, serverVersion?}]}): renders a version history inside the + editor, with per-version change data when changes+serverVersion match; and + setHistoryData({url, key, previous{...}, changesUrl}): links for viewing a + stored version. Also createConnector() (Automation API, callable in + onDocumentReady), requestClose(), denyEditingRights(message). Consequences: + an external revision can reach a doc that is open in someone's editor + WITHOUT a full page reload -- the building block of live co-editing (§5.7); + our stored revisions + changes.zip archives can power the editor's own + version-history UI. Programmatic push (calling refreshFile outside the event) and whether + client-side config events are reachable from our page (same-origin or via a + postMessage bridge): M0 territory. +- **Comment events**: the current official callback body does **not** include + commentadd/commentremove push events (no events array in the documented + parameters). Comment wake is therefore implemented server-side from the saved + file (§5.4) -- this supersedes the issue's assumption that DS pushes comment + events to the callback. Client-side config.events.onCommentAdd exists (runs + inside the editor page) and can later feed a postMessage bridge for live UI, + but it is not the wake path. + +## 5. Component design (server half) + +### 5.1 Lifecycle -- src/host.ts + src/container.ts + +- host.ts: module-level let current: ServerPortal; setPortal(portal), portal(), + docker() accessors (understory shape). Default export of server entry: set + portal -> register routes -> void settleKept().catch(log) in background + (adopt container + volume if they exist) -> setInterval(lookAfter, + 30_000).unref(); cleanup clears the timer. +- Container spec (Docker Engine ContainerCreate body): + - image: **pinned tag** onlyoffice/documentserver:<X.Y> -- exact tag chosen in + M0 spike; never :latest (image changes must not silently alter callback/jwt + behavior). + - env/volume per JWT mechanism from M0; named volume for /var/www/onlyoffice/Data. + - network: await docker.sharedNetwork() HostConfig mode ("host" or + container:<portal-id>), so the container reaches the loopback listener for + document.url/callback. Under shared networking, Docker PortBindings are + IGNORED and there is no published port to persist: DS stock nginx simply + listens on :80 of the **all interfaces of the shared netns** -- in host + mode that is the host's 0.0.0.0:80 (collision risk if anything else binds + it, plus LAN/tailnet exposure). There is no documented listen-address env + for DS. The DS origin is fixed by deployment shape (127.0.0.1 in host + mode; the container-local equivalent otherwise) and must be recorded in + ds.json on create. + - Browser reach to that origin is a **decision, not an assumption** -- see + M0 spike 2 (in-container nginx bind override vs portal reverse-proxy route + vs declared local-only access). v1 ships with whatever M0 picks; the plan + no longer claims loopback-only publication. +- Settings (dataDir/settings.json): {enabled, imageTag}. Start/stop + from settings page; lookAfter keeps a user-started container alive but never + resurrects one the user stopped. Status route reports: container state, image + pull progress (images.ensure onProgress callback, held in memory), DS version + (exec into container). + +### 5.2 Documents & tickets -- src/store.ts + src/tickets.ts + +- Documents CRUD in the extension's own library (`portal.dataDir()`): layout + docs/<docId>/<title>.<ext> + index.json entries {docId, title, ext, + createdAt, updatedAt, sizeBytes, mtime(revision), owner: chatId|null, + grants:[{chatId,commentId,at}], createdBy}. Extension filter: .docx/.xlsx (.pptx later). All reads/writes are + plain node:fs inside the extension's own directory -- the verified 1 MB cap + (MAX_EDIT_BYTES, core/server/src/workspace-files.ts:47) applies to + portal-managed files, not ours, so >1MB Office files work natively here. + Persist stays write-to-temp + rename with an explicit mtime conflict gate + (|mtimeMs - expected| > 1 ms => refusal); delete = move to .trash/ with a TTL. +- **Ownership/grants gate agent access**: home-chat panels, tool registration + in the chats.extend factory (tools register only in agent/project homes), + and office_read/office_edit checks. Human UI routes are unrestricted; the + loopback GET /doc/<ticket> stays HMAC + expiry only (DS-facing, no user + context). +- Import/export to chat folders: copy in/out via portal.files.open() fd + fs; + import performed from an agent/project context sets that home as owner, else + ownerless (keeps Files-extension users and "send this doc to my chat" + working). +- Blank templates: minimal valid docx/xlsx generated with a small zip library + (no zip/jwt/docx dep exists in the monorepo lockfile today -- add one pure-JS + zip lib, proposed jszip; everything else is node:crypto/https/fs). +- **Cookie-free container endpoints (design decision, not open)**: verified in + core that EVERY portal route is session-gated -- app.use("/api", requireAuth) + (core/server/src/server.ts) and extension routes mount at /api/x *after* the + login (extension-host; packages/extension-api/server.d.ts). The DS container + holds no session cookie, so a ticketed GET AND the callback POST would both + 401 if registered on portal.router. Design: a tiny standalone http listener + in the same process, bound to **127.0.0.1 only**, serving exactly two paths: + - GET /doc/<ticket> -- mint per-open HMAC-SHA256 tickets: token = + base64url(docId|expMs|hmac(secret, docId+exp)), 10 min expiry, verified + with timingSafeEqual; answers the library file's bytes (plain fs, §5.2 -- + no portal files size cap on our own dir). This is document.url in the + minted config. + - POST /callback -- handled per §5.3 (DS -> listener direct; nothing passes + through the login middleware). + sharedNetwork() puts the container on this loopback in BOTH deployment + shapes (host: the host's; container: the portal container's), so one design + covers both. No other endpoint lives on the listener. + +### 5.3 Callback & lock state machine -- src/callback.ts + +Served by the loopback listener (§5.2), not a portal route. Verify JWT +(secret; reject with error code if invalid -> answer {error:1}), then branch +on status exactly per §4: + +| status | lock | persist | notes | +|---|---|---|---| +| 1 | set users[] from body (lock = users.length>0) | -- | actions tell join/leave/forcesave; emit UI event | +| 2 | unlock after persist | fetch url -> temp file in dataDir -> fs write+rename over the library file with the mtime gate of §5.2 | bump revision (mtime); store changes.zip from changesurl under dataDir/history/<doc>/<rev>.zip; run comment-diff wake (§5.4) | +| 3 | keep | log + audit.record | answer {error:1} only if we also failed internally | +| 4 | unlock | -- | no change on disk; comment diff still runs cheaply on next persist | +| 6 | keep locked | same persist as 2 without unlock | timer/Save-button forcesave while session continues | +| 7 | keep | log + audit.record | force-save error | + +Persist must be resilient: retry fetch a few times (DS can lag), never delete +the existing file on failure (temp+rename), and the file's mtime/sha is the +revision tools compare against. document.key includes the revision hash so DS +never serves stale cached content after an agent edit. + +### 5.4 Wake on comment mention -- src/ping.ts (server-side) + +On every persist (status 2/6): unzip the saved docx, parse word/comments.xml +({id, author, date, text = joined w:t}), diff against the last-seen comment set +stored in dataDir/state/<doc>.json. For each **new** comment, resolve its mentions (grammar below), then per +mention: + +**Mention grammar (user decision)** -- parsed from the comment text: +- `@agent` (literal token) -> the document's **owner**, i.e. the agent/project + that created it; ownerless docs: ignored + logged. +- `@project:<name>` -> the project with that display name (case-insensitive + exact match). +- any other `@<name>` -> the agent with that display name; its work starts in + that agent's home chat. +- Unresolvable/duplicate names: no wake, logged (risk item -- no silent + disambiguation). + +**Routing + grant (the permission model):** +1. Resolve mention -> target chatId (home of the named agent/project). Create + a grant if absent: doc.grants += {chatId, commentId, at} -- mentioning is + what authorizes that entity for this file; grants persist until revoked + from the UI (D3), are scoped to this one doc, audited. +2. Queue wake entry for the target chat: {text: officeWakeMessage(docTitle, + author, commentText), details:{docId, commentId}}, deduped by comment id + per chat. +3. portal.chats.tell(targetChatId, {type:"ping"}) (rings UI_RECEIVE bell; text stays queued server-side -- exact docs pattern). +4. The target home's pi extension (registered from the same chats.extend + factory) listens on UI_RECEIVE and drains via + pi.sendMessage({customType:"office-comment-ping", content, display:true, + details}, {deliverAs:"followUp", triggerTurn:true}). + +When the M0 client-API spike confirms client-side event access (same-origin or +via a postMessage bridge), a new mentioning comment can wake immediately +instead of after the next persist; the save-based diff stays the guaranteed +fallback (both dedupe by comment id and create grants exactly once). + +Wake message shape follows docs' wakeMessage(): doc title, quoted comment, +doc id, and "you have edit access to this document for this task; answer with +office_* tools". +v1 wake targets **docx only**: xlsx does have comments, but they live in a +different part layout (xl/commentsN.xml + VML anchors + sheet rels), not +word/comments.xml -- parsing that layout is deferred to M4 (plan wording +corrected: it was "xlsx has no comments.xml", which is false). + +### 5.5 Agent tools -- src/tools.ts (chats.extend) + +Docs-style: id = relative path, writes carry baseRevision (= mtime), refusals +are Errors prefixed with codes. Revision gate (mtime conflict) on every +mutating tool; an open human session is NOT a refusal -- forcesave-first +ordering per §5.7 (co-editing default ON). + +- office_list -> [{id, title, ext, sizeBytes, mtime, locked, users}] +- office_create {title, type:"word"|"cell"} -> {id} (from template; owner = this home) +- office_read {id} -> extracted text (paragraph-delimited for docx; grid of cell values for xlsx) + comments [{id, author, date, text}] + lock state. Word: word/document.xml runs; XLSX: sheet XMLs with sharedStrings. +- office_edit {id, ops:[{quote, replacement}], baseRevision} -> paragraph/run level replacement in document.xml (docs' quote philosophy: quote_not_found / quote_ambiguous), rewrite zip preserving every other entry byte-for-byte. Safety against a live session is ordering, not refusal (§5.7): a dirty human session is forcesaved first (status 6 persists before our write); otherwise the edit is queued until all users closed; baseRevision is checked either way, so a stale edit still fails with revision_mismatch. +- office_comments {id} -> read-only thread list (v1). + +Stretch (M4): office_comment_add -- injecting a comment requires editing +word/comments.xml + body ranges + [Content_Types].xml/rels; high corruption risk +to user files. Only with a strict round-trip validator (refuse on any schema +surprise); otherwise agent replies stay in chat + text edits. + +Tool registration is ownership-scoped: office_* register only in agent/project home chats, and there they operate only on docs where that home is **owner** or holds a **grant** (§5.4), so office_list returns exactly its documents and office_create sets owner to the home (confirmed D1). With co-editing default ON (§5.7) an open human session no longer refuses -- it triggers forcesave-first ordering; the revision gate (baseRevision) always remains. + +### 5.6 Security + +- JWT secret: ONE crypto.randomBytes(32) value, generated once, stored 0600 in + dataDir (extension's own 0700 dir), passed to DS as JWT_SECRET and used for + both config signing and callback verification (see §4 -- the community image + exposes no per-direction env vars); constant-time compare. +- Listener endpoints (/doc/<ticket>, /callback) bind 127.0.0.1 only -- that is + under our control. The DS web origin (:80 in the shared netns) is NOT + loopback-only by construction; its exposure is handled by the M0 spike 2 + decision and documented on the settings page, not claimed away. +- Ticketed document route: expiry + HMAC only (DS-facing; user-level access + is gated at config mint and in the tools, §5.2). No other auth bypass. +- audit.record() on: container start/stop, document create/rename/delete, + every grant + revoke, import/export, every persist, every agent edit, + every wake (subject=doc id, chatId set). + +### 5.7 Live agent co-editing (default ON -- user-confirmed; disable setting as escape hatch) + +There is no official API to stream external edits into DS's live session state +(DS owns the document once a session opens and only ingests new content via a +key/config change), so "live" is built from the documented building blocks in +§4. Fidelity tiers, best first: + +1. **Refresh tier (v1, the default behavior)**: office_edit succeeds on a doc + with an open human session -> persist new revision + store changes snapshot + -> mint fresh config (new document.key + ticket url) -> deliver it to every + same-origin editor showing the doc via DocEditor.refreshFile(config) + (programmatic push -- M0 verifies DS honors it outside + onRequestRefreshFile; fallbacks: wire the editor UI's own update-request + event, which is the documented path, or badge "agent updated this document + -- refresh" for the human). Unsaved human changes are never clobbered: if + the session reports unsaved state (appstate), trigger a forcesave first and + apply after its status-6 persist, else queue until clean. +2. **History tier (v1 always)**: every agent revision is registered via + refreshHistory({history:[...stored changes.zip + serverVersion...]}) so it + appears in the editor's own version-history UI with a diff -- true "real + office" feel even when live push is off. +3. **Connector tier (M0 spike only)**: createConnector() (Automation API) in + onDocumentReady -- if the client-side connector can apply text operations + to the open session, agent ops stream into the live document with no + refresh at all. This is the only path to true character-level co-editing; + it depends on behavior the docs treat as secondary, so it stays a spike, + not a commitment. + +## 6. Component design (browser half) -- web/ + +Copy the docs extension's client structure: + +1. **Global nav.page** {id:"office", title:"Office"} -- the full library at + /x/pithagoras-office/office, deliberately styled like an office suite with + its own minimal chrome (user direction: pithagoras card style does not fit + an office suite): thin toolbar (container start/status + pull progress when + stopped; version when running), document list/grid of ALL docs (title, ext + icon, owner badge, grant badges, mtime, lock badge from /state polled every + ~5 s), create button, row menu: open, import/export to chat folder, revoke + grants, delete. Row click or ?doc= -> full-bleed OfficeEditor in the same + tab (thin header only; docs precedent: DocumentsPage switches on ?doc=). + If M0 leaves browsers on the direct DS origin, also offer "open at DS" as a + plain link. +2. **Per-home chat.panel** "office" (agent/project homes): compact list of the + docs this home owns or has grants on + lock badges; onExpand -> navigate to + the editor page with ?doc=. +3. **OfficeEditor**: header (title, back-to-list, live "N editing..." lock + badge) + the editor placeholder that DocEditor fills (DS creates its own + iframe). Page requests POST /editor-config {doc}; server mints the + JWT (+ mode/permissions per the regulated signing set); a placeholder div is + filled by new DocsAPI.DocEditor(el, config) -- the documented api.js pattern + (§4), with document.url and callbackUrl pointing at the loopback listener. + When the M0 client-API spike confirms the push surface (same-origin or via + postMessage bridge), this page additionally calls: refreshFile() delivery + for co-editing (§5.7), refreshHistory()/setHistoryData() for stored agent + revisions, and appstate/config events for an instant comment-mention wake + plus a no-poll lock badge. +4. **chat.tool** entries per office_* tool: result card with doc title and an + "Open in Office" button (docs tool-card pattern). +5. **settings.page** "office": enable/disable, image tag display (pinned), + start/stop, status block, RAM warning (~4 GB recommended) shown before the + first start, data locations, co-editing toggle. +6. style.css: lock badge + grant badges, office-suite list chrome for the + global tab (deliberately not portal card style). + +## 7. Tests + +- **Unit** (extensions/office/test/*.test.mts, run by the root + npm test glob node --import tsx --test ... extensions/*/test/*.test.mts): + ticket mint/verify/expiry; state machine over synthetic callback sequences + (join -> leave-with-changes: file updated + unlocked; forcesave keeps lock; + invalid JWT rejected; temp+rename atomicity); docx fixture round-trip + (extract text; quote replace preserves other zip entries byte-for-byte; + comments.xml diff detects exactly the new mentioning comment, dedupes by id); + ownership/grant scoping (doc owned by home A absent from home B's tools/list; + a mention-grant makes it editable for B; ownerless docs invisible to all + agents); + revision-mismatch refusal in office_edit; live-mode ordering (dirty session + -> forcesave scheduled before refreshFile config mint). +- **Integration** (docker-gated, skip when docker.available() is false): + container boots from image (CI-cached); the loopback listener answers a + ticketed document GET and a synthetic callback **without any session cookie** + (HMAC + expiry enforced); a document >1 MB + round-trips persist in the library store; editor page loads the + DS editor via DocEditor (or the portal proxy route if M0 picks that) for a + minted config; full save loop with a scripted callback + real DS-hosted url + fetch. +- **Playwright** (test/browser/office.spec.mts, repo's @playwright/test setup): + office page lists documents; start-container flow shows progress then badge; + opening a doc renders the DS editor in the page (DocEditor, or proxy route if + M0 picks that) with no error banner; settings start/ + stop round-trips. Skipped when docker is unavailable in the test env. + +## 8. File layout (new) + + extensions/office/ + package.json # pithagoras-office, keywords ["pi-package"], "pi": {}, + # "pithagoras": {client, server, style, + # requires: {"pithagoras-docker": "^1.1.0"}} # nested -- core reads it there + build.mjs # esbuild from repo node_modules (docs pattern) + deps.js # missingDependencies -> npm ci --ignore-scripts + src/host.ts # portal registry + 30 s lookAfter loop + src/container.ts # create/adopt/stop, DS origin recording (ds.json), status + src/settings.ts # dataDir settings.json + src/jwt.ts # HS256 sign/verify (node:crypto) + src/tickets.ts # HMAC download tickets + src/store.ts # library in dataDir: index.json, docs/<docId>/, .trash/; + # owner + grants; import/export to chat folders; templates (jszip) + src/live.ts # live co-editing: refreshFile/history delivery, forcesave-first ordering + src/callback.ts # state machine + persist + lock events + src/ooxml.ts # docx/xlsx text+comments parse, quote-level edits + src/ping.ts # comment diff -> wake queue (docs ping pattern) + src/tools.ts # office_* tools, lock/revision gates + src/server.ts # default export: routes wiring everything above + web/client.ts # nav.page, panel, tool cards, settings page + web/office-editor.ts # header + api.js/DocEditor mount + style.css + test/*.test.mts # unit + docker-gated integration + test/browser/office.spec.mts + fixtures/sample.docx, sample.xlsx + +No changes to core/*, no new root workspaces entry, no change to defaults.json. + +## 9. Milestones + +**M0 -- spikes (answer the unknowns; ~1-2 d)** +- Reachability: container -> loopback listener (ticket + callback) under + sharedNetwork() in BOTH host mode and container mode; record the exact + DS-side origin for document.url/callbackUrl and the browser-side DS origin; + confirm a port-80 collision is surfaced clearly when starting. +- Browser reach to DS: three options -- (a) keep direct DS origin and declare + editor access local-machine-only with documented exposure; (b) spike an + in-container nginx override so DS actually listens on 127.0.0.1:<P> inside the + container (fragile vs image drift); (c) portal reverse-proxy route for DS + traffic (portal.http.raw + portal.http.upgrade, already login-checked) with + the iframe on the portal origin -- remote users work, DS stays server-side, + and same-origin unlocks the programmatic DocEditor surface of §5.7 (c is the + preferred target; fall back to a/b only if the proxy spike fails). Pick one; + this decides §6 OfficeEditor's src mechanism. +- Same-origin client API surface (assumes option c): does + DocEditor.refreshFile(newConfig) called programmatically -- not in answer to + onRequestRefreshFile -- update an open editor? do client-side config events + (comment add, appstate) fire for us same-origin? does createConnector() + (Automation API) expose a usable client-side surface on the live session? + Result decides the §5.7 fidelity tier shipped in M2/M3. +- JWT configuration of the pinned image: confirm single JWT_SECRET env covers + config + callbacks (expected from docs); only if a supported per-direction + knob exists on that build consider the two-secret variant. Also pick the pin + (not :latest), verify status semantics match §4, and capture the exact + regulated signing set for the config token; measure idle RAM. +- Blank docx/xlsx template validity: open in real editor, save back cleanly. +Deliverable: spike notes appended to this file; plan amendments where needed. + +**M1 -- human editor works** +Package skeleton + build; lifecycle (start/stop/status/settings page); library +store (index.json, docs/<docId>/, owner + grants, import/export to chat +folders); verify dataDir persistence/backup treatment; storage host routes; +**global Office tab** (library browser) + per-home panels; OfficeEditor iframe opens a document and edits it; +save round-trip lands the updated .docx in the library with bumped revision; +atomic persist verified. Acceptance: create -> open -> edit -> close -> file on +disk changed, lock released, status page clean. + +**M2 -- agent tools** +office_list/create/read/edit (+ xlsx read), lock + revision gates, coded errors, +audit entries, tool cards in chat UI. Acceptance: unit tests green; agent edits +a docx and the change reaches the open editor via the refresh flow (§5.7, +key rotation works) or renders on next open when no session is live. + +**M3 -- wake on mention** +comments.xml diff, **mention grammar** (@agent = owner / @<name> = agent / +@project:<name> = project), home-chat routing, grant creation + audit, ping +queue + per-home pi delivery, panel + background entries (docs pattern). +Acceptance: a human comments "@aegis ..." in an open doc -> after save/close +(or forcesave) Aegis's home fires the turn with the quoted comment AND gains a +visible, audited grant on that doc; bare @agent wakes the owning agent/project; +no double-wake on repeated callbacks. + +**M4 -- polish & stretch** +Co-editing hardening (connector tier if the spike won; strict per-comment +grant expiry per D3), xlsx cell edits + xlsx comment wakes +(xl/commentsN.xml layout), office_comment_add (only +with strict validator), postMessage live lock badge, .pptx read, docs page in /docs, fresh-portal opt-in install +test, RAM warning UX. Done = issue #280 ready to close. + +## 10. Risks / open items + +- **DS build drift**: callback/jwt behavior changes across major versions -- + hence pinned tag + M0 verification + integration test. +- **OOXML edit fragility**: run-splitting in w:t is the classic corruption + vector; mitigations: paragraph-level targeting, byte-preserving zip rewrite, + fixture round-trip tests, refuse-on-surprise. +- **Resource floor**: ~4 GB RAM (issue-accepted trade-off); surfaced at enable + time; status page shows container memory via inspect. +- **Container-endpoint plumbing**: the cookie-free endpoints sit on a + standalone loopback listener BY DESIGN (verified: every /api/* route is behind + session auth, so a portal router route would 401 for DS). Least certain part: + the browser-reach decision (M0 spike 2) and port-80 behavior of shared + networking in both deployment shapes. +- **Co-editing is a refresh model, not true co-editing** unless the + connector spike lands. It is now the DEFAULT (user-confirmed), so the + forcesave-first ordering against unsaved human changes is critical safety + logic; a disable setting remains as escape hatch. +- **Mention name resolution**: agents/projects matched by display name + (case-insensitive exact). Duplicate names -> no wake + log; enforcement or + disambiguation is polish, not v1. +- **dataDir persistence/backup**: assumed to match chat-folder treatment; M1 + verifies, otherwise the library location or a docs note moves. +- **Spreadsheet wake**: xlsx stores comments differently (xl/commentsN.xml + + VML anchors, not word/comments.xml) -- v1 docx-only; deferred, not a blocker. + +## Appendix A -- verified references + +- Callback handler reference: https://api.onlyoffice.com/docs/docs-api/usage-api/callback-handler/ + (fetched; status table + parameters + Node.js save example as summarized in §4) +- Config reference: https://api.onlyoffice.com/docs/docs-api/usage-api/config + (documentType values, token field, embedded type) +- DocEditor embedding + regulated signing: + https://api.onlyoffice.com/docs/docs-api/usage-api/doceditor/ and + .../additional-api/signature/browser (fetched; api.js entry point, + placeholder -> iframe replacement, signed parameter set incl. + document.permissions / editorConfig.mode since 7.1) +- Client-side DocEditor methods (fetched): + https://api.onlyoffice.com/docs/docs-api/usage-api/methods/ -- + refreshFile("updates the file version without reloading the editor", + documented via onRequestRefreshFile), refreshHistory/setHistoryData (version + history with changes + serverVersion), createConnector (Automation API), + requestClose, denyEditingRights +- Docker deployment: helpcenter community-install + + github.com/ONLYOFFICE/Docker-DocumentServer README (env vars only + JWT_ENABLED/JWT_SECRET/JWT_HEADER/JWT_IN_BODY on the 10.x community image; + local.json regenerated on restart; system requirements RAM >= 4 GB, DS on :80, + no listen-address env; current tags 10.0/10.0.0) +- In-repo checks for the second/third revisions: portal.dataDir() exists + (packages/extension-api/server.d.ts:844; understory store precedent); + chat kind is "agent"|"project"|"chat" (server.d.ts:95) so per-agent/project + ownership maps to the owner/grants index fields. +- In-repo checks done for the first revision (technical review): sharedNetwork() returns "host" or container:<id> (extensions/docker/src/docker.ts:33-44); a shared-netns container publishes no ports because Docker ignores PortBindings under host/container network modes (Docker semantics, not stated in the repo file); portal.files 1 MB cap + (core/server/src/workspace-files.ts MAX_EDIT_BYTES); /api/* session-gated + (core/server/src/server.ts requireAuth; extension-host mounts at /api/x after + login); pithagoras.requires nesting (extension-host loader; understory + package.json) +- docspace-mcp: https://github.com/ONLYOFFICE/docspace-mcp -- README verified: + Docspace cloud SaaS target (DOCSPACE_BASE_URL / DOCSPACE_API_KEY), toolsets = + rooms/folders/permissions/people; NOT an on-prem DS file-content MCP. +- In-repo precedents cited in §3 table (understory host/server, docker api.ts, + docs client/tools/ping). +- xlsx comment parts (xl/commentsN.xml) per the OOXML/ECMA-376 spreadsheet + packaging structure. diff --git a/core/server/test/extension-ui.test.mjs b/core/server/test/extension-ui.test.mjs index 9b8d77ff0..72047ff6c 100644 --- a/core/server/test/extension-ui.test.mjs +++ b/core/server/test/extension-ui.test.mjs @@ -213,7 +213,7 @@ test("of two installed under one name, the one switched on is the one a plan fin test("what comes with the portal is listed to install from its own folder, without the subagent tool, which has a switch of its own in Settings → Tools", async () => { const { extensions } = await (await fetch(at("/api/extensions/shipped"))).json(); - assert.deepEqual(extensions.map((e) => [e.name, e.kinds]), [["pithagoras-browser", ["pithagoras"]], ["pithagoras-canvas", ["pithagoras"]], ["pithagoras-creator", ["pithagoras"]], ["pithagoras-docker", ["pithagoras"]], ["pithagoras-docs", ["pithagoras"]], ["pithagoras-files", ["pithagoras"]], ["pithagoras-git", ["pithagoras"]], ["pithagoras-kanban", ["pithagoras"]], ["pithagoras-media", ["pithagoras"]], ["pithagoras-terminal", ["pithagoras"]], ["pithagoras-themes", ["themes"]], ["pithagoras-understory", ["pithagoras"]], ["pithagoras-voice", ["pithagoras"]], ["pithagoras-voice-servers", ["pithagoras"]]]); + assert.deepEqual(extensions.map((e) => [e.name, e.kinds]), [["pithagoras-browser", ["pithagoras"]], ["pithagoras-canvas", ["pithagoras"]], ["pithagoras-creator", ["pithagoras"]], ["pithagoras-docker", ["pithagoras"]], ["pithagoras-docs", ["pithagoras"]], ["pithagoras-files", ["pithagoras"]], ["pithagoras-git", ["pithagoras"]], ["pithagoras-kanban", ["pithagoras"]], ["pithagoras-media", ["pithagoras"]], ["pithagoras-office", ["pithagoras"]], ["pithagoras-terminal", ["pithagoras"]], ["pithagoras-themes", ["themes"]], ["pithagoras-understory", ["pithagoras"]], ["pithagoras-voice", ["pithagoras"]], ["pithagoras-voice-servers", ["pithagoras"]]]); for (const shipped of extensions) { assert.ok(path.isAbsolute(shipped.spec) && shipped.spec.endsWith(path.join("extensions", shipped.name.replace(/^pithagoras-/, ""))), shipped.spec); assert.ok(shipped.version && shipped.description, shipped.name); diff --git a/extensions/office/.gitignore b/extensions/office/.gitignore new file mode 100644 index 000000000..061f4e25f --- /dev/null +++ b/extensions/office/.gitignore @@ -0,0 +1,4 @@ +# Built by build.mjs (the portal's npm run build), not kept in git. +/web/ +/server.js +/node_modules/ diff --git a/extensions/office/README.md b/extensions/office/README.md new file mode 100644 index 000000000..17feaaf9e --- /dev/null +++ b/extensions/office/README.md @@ -0,0 +1,42 @@ +# pithagoras-office + +Office documents (.docx and .xlsx) edited in a real [OnlyOffice Document Server](https://github.com/ONLYOFFICE/Docker-DocumentServer) container that the portal runs of its own. The plan of record is `PLAN-office-280.md` (issue #280); this note says what ships in v1 and where everything lives. + +## What it is + +An opt-in package (it requires `pithagoras-docker`). In plain terms, v1 gives you: + +- **A document library** on the portal's Office tab: create blank Word/sheet documents, open them full-screen in the embedded OnlyOffice editor, sort the list by name / modified / size / type, and export (download) a copy of one. There is no manual import: every `.docx`/`.xlsx` found across the agent homes and projects (`portal().folders.list()`) is indexed **automatically at boot** (and on demand with the **Rescan workspace** button) — IN PLACE, so the scanned file IS the document and is edited through the same mtime-gated flow. Deleting a workspace-scanned document moves its real file into this library's trash, like everything else: reversible for 7 days. +- **Agent tools** (`office_list`, `office_create`, `office_read`, `office_edit`, `office_comments`) registered in agent/project home chats. Agents read and edit the SAME files humans edit; writes are quote-based replacements gated on a revision number (the file's mtime), so a stale agent edit fails loudly instead of clobbering a human save. +- **Co-editor model (no switch)**: the agent is just another user in the editor, like Office 365 — a human and the agent may edit the same document at the same time, and there is no setting to turn that on or off. Agent edits always proceed under the strict mtime/revision gate; if the file changed since they were read (for example a co-editor's save), NOTHING is written and the tool answers a loud `error:revision_mismatch` CONFLICT naming the open editor session when one stands — re-read and retry. Conflicts are surfaced, not prevented by refusal. Open editors learn about external changes through the existing 5 s polling loop (`live.ts pollState`). Every agent revision is recorded for the editor's version history. +- **Wake on mention**: when a human comments `@agent` (the doc's owner) or `@<name>` / `@project:<name>` (a named agent/project) and saves, that home chat gets pinged with the quoted comment and gains an audited grant on the document. + +## Requirements + +- Docker on the host the portal runs from (shared networking: the DS container joins the same netns as the portal). +- **~4 GB of RAM** for the Document Server container — the settings page warns before first start, and the status block shows live memory usage once it runs. + +## Setup + +1. Install/enable the package, then open Settings → office: enable it and press start. The pinned image (default `onlyoffice/documentserver:10.0` — never `:latest`; changing the tag can silently change callback/JWT behavior) is pulled on first start with progress shown in the UI. +2. Documents appear under the Office tab: what you create here, plus every `.docx`/`.xlsx` found across the agent homes and projects — indexed automatically at boot (the scan runs after `settleKept`, so files that vanished while the portal slept are pruned, not re-added), with Rescan picking up anything written straight to disk afterwards. Open one to edit; agents work on it through their chat tools. + +## Where things live, and what is exposed + +All state is inside the extension's own data directory (`portal().dataDir()`): `settings.json`, `jwt-secret` (0600, created once), `ds.json` (container id + recorded origin), `index.json` + `docs/<docId>/<title>.<ext>` (the library), `.trash/`, `state/<docId>.json` (last-seen comment ids), and `history/<docId>/<rev>.zip` (change snapshots for the editor's version history). + +The DS container itself holds no session cookie, while every portal route is login-gated — so the two endpoints the container needs (a ticketed document GET and the callback POST) live on a **standalone loopback listener bound to 127.0.0.1 only**, on an ephemeral port minted per boot: `GET /doc/<ticket>` (HMAC + 10-minute expiry, no user context) and `POST /callback` (HS256-verified). Nothing else listens on it, and nothing outside the container's own network can reach it. The browser-facing half of the DS goes the other way: a **portal reverse proxy** (`GET /ds/<rest…>` under this extension's session-gated routes) forwards the editor traffic — api.js, the editor iframe, every asset — to the recorded DS origin, so remote browsers reach the editor through exactly the same URL prefix they use for the portal. That is why `document.url` and `editorConfig.callbackUrl` can stay absolute loopback URLs in the signed config: only the DS server itself fetches them. + +## Deferred (honest list) + +From the plan, intentionally NOT in v1: + +- **xlsx comments** (`xl/commentsN.xml` + VML layout) — reading and wakes for spreadsheet comments are M4; Word comments work now. +- **Co-editing refresh handshake**: whether `DocEditor.refreshFile()` can be pushed programmatically was an M0 spike; v1 ships with the "agent updated — refresh" badge fallback. The M0 browser-reach question resolved to the portal reverse-proxy route (`/ds`, see Security notes) — the editor surface is same-origin. +- `.pptx` reading, `office_comment_add`, per-comment grant expiry, postMessage live lock badges — M4 stretch items. +- **Exporting into chat folders** — the export route serves a download of the stored bytes (fresh copy, served as an attachment); copying a document into a chat folder does not exist in v1. + +## Security notes + +- **The `/ds` proxy stays behind the session gate.** Browser editor traffic (api.js, iframe, assets) reaches the DS only through `GET /api/x/<extension>/ds/…`, mounted like every other route of this extension behind the portal's login — and it forwards to exactly one origin: the recorded DS origin in `ds.json` (`"container:<id>"` markers refuse with 422, an unreachable origin with 503). GET-only, no body, single destination. +- **Host networking exposes the Document Server on every interface.** The DS container runs with the portal's shared network (`NetworkMode` from `docker.sharedNetwork()`; this deployment resolves it to `"host"`), so the image's own nginx serves the DS web origin (:80) on 0.0.0.0 — anything that can reach that machine's port 80 (the local host, the LAN, a tailnet peer) reaches the Document Server UI and API without going through any portal login gate: v1 has no authentication in front of that origin (the per-config JWTs sign requests the PORTAL mints; they are not an access-control layer for strangers). Operators should firewall :80 accordingly (e.g. reachable from the machine itself only), or override `/etc/nginx/conf.d` via a named volume to bind nginx to loopback — planned work, not something v1 does. diff --git a/extensions/office/build.mjs b/extensions/office/build.mjs new file mode 100644 index 000000000..ca39d318d --- /dev/null +++ b/extensions/office/build.mjs @@ -0,0 +1,27 @@ +/** + * Builds pithagoras-office: its browser half (src/client.tsx) into web/, with the Tailwind utilities its components use + * made into its own stylesheet from the portal's preset, and its server half (src/server.ts) into server.js. The + * portal's own `npm run build` runs this, so a checkout and the image build it the same way; what it makes is not kept + * in git. + * + * What it is built and runs with (react-icons for the browser half, jszip for the OOXML templates on the server + * side -- which deps.js ALSO installs at boot in the background, as pithagoras-docs does) is this package's own, + * installed here first when it is not there yet, or not at the version its lock file names (missingDependencies, + * shared with deps.js). esbuild, tailwindcss and postcss are the repository's. + */ +import { execFileSync } from "node:child_process"; +import path from "node:path"; +import { buildExtension } from "@pithagoras/extension-api/build"; +import { missingDependencies } from "./deps.js"; + +const HERE = import.meta.dirname; + +// Missing, or not the versions the lock names (a later commit changed them): installed as the lock says. +const missing = missingDependencies(HERE, { dev: true }); +if (missing.length) { + console.log(`pithagoras-office: installing its packages (${missing.length})`); + execFileSync(process.platform === "win32" ? "npm.cmd" : "npm", ["ci", "--ignore-scripts", "--no-audit", "--no-fund", "--prefer-offline"], { cwd: HERE, stdio: "inherit" }); +} + +await buildExtension({ dir: HERE, tailwind: true, server: "src/server.ts" }); +console.log("pithagoras-office: built web/ and server.js"); diff --git a/extensions/office/deps.d.ts b/extensions/office/deps.d.ts new file mode 100644 index 000000000..9f426d1ab --- /dev/null +++ b/extensions/office/deps.d.ts @@ -0,0 +1,4 @@ +/** Type surface of deps.js (its plain-JS twin beside this file): what the built server imports to install this + * package's own node_modules at boot, in the background. esbuild bundles the .js; tsc reads these declarations. */ +export function missingDependencies(dir?: string, opts?: { dev?: boolean }): string[]; +export function ensureDependencies(log: (line: string) => void, opts?: { dir?: string; npm?: string }): Promise<boolean>; diff --git a/extensions/office/deps.js b/extensions/office/deps.js new file mode 100644 index 000000000..1671a3e33 --- /dev/null +++ b/extensions/office/deps.js @@ -0,0 +1,94 @@ +/** + * The extension's own dependencies, there before its server half needs them. + * + * pi installs a package by its path without installing what the package + * depends on, and the image installs them when it is built. A folder that + * comes without them (a checkout the portal runs from, a copy made by hand) + * has them installed here, once: jszip, for the minimal docx/xlsx templates + * its store writes, which the server bundle imports from its own node_modules + * at runtime (buildExtension leaves packages external there). The install is + * in the background, since a server half has a few seconds to start and an + * install takes longer. + */ +import { spawn } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); + +function readJson(file) { + try { + return JSON.parse(readFileSync(file, "utf8")); + } catch { + return null; + } +} + +/** + * What the package in `dir` needs and does not have in its own node_modules as + * its lock file says: missing, or at another version (a later commit changed + * the lock, and nothing installed it since). Only what it needs to run, unless + * `dev`: then also what it is built with. Named as npm names them. + */ +export function missingDependencies(dir = HERE, { dev = false } = {}) { + const lock = readJson(path.join(dir, "package-lock.json")); + const wrong = new Set(); + if (lock?.packages) { + for (const [where, entry] of Object.entries(lock.packages)) { + if (!where.startsWith("node_modules/") || entry.link || (entry.dev && !dev)) continue; + const installed = readJson(path.join(dir, where, "package.json")); + // An optional package is one npm leaves out where it does not fit (another system): only one that is there is checked. + if (!installed && entry.optional) continue; + if (installed?.version !== entry.version) wrong.add(where.slice(where.lastIndexOf("node_modules/") + "node_modules/".length)); + } + return [...wrong]; + } + // No lock: what the package names, there at all. + const meta = readJson(path.join(dir, "package.json")) ?? {}; + return Object.keys({ ...meta.dependencies, ...(dev ? meta.devDependencies : {}) }).filter((name) => !existsSync(path.join(dir, "node_modules", name, "package.json"))); +} + +/** + * One install at a time per folder, for as long as the portal runs: kept for + * the process, since this module is read again with each revision of the + * server half, and an install a revision before started goes on after it. + */ +const installs = (globalThis[Symbol.for("pithagoras-office/installs")] ??= new Map()); + +/** + * Installs what is missing or out of date with npm, as the lock file says, and says so in the + * portal's log. Resolves true once everything is there, false when the install + * failed (and a later start tries again). + */ +export function ensureDependencies(log, { dir = HERE, npm = process.platform === "win32" ? "npm.cmd" : "npm" } = {}) { + const missing = missingDependencies(dir); + if (!missing.length) return Promise.resolve(true); + if (installs.has(dir)) return installs.get(dir); + log(`Installing its dependencies (${missing.join(", ")}) with npm`); + const started = Date.now(); + const install = new Promise((resolve) => { + // Only what it needs to run, and without install scripts: jszip is plain JS. + const child = spawn(npm, ["ci", "--omit=dev", "--ignore-scripts", "--no-audit", "--no-fund"], { cwd: dir, stdio: ["ignore", "ignore", "pipe"] }); + let said = ""; + child.stderr.on("data", (chunk) => (said = (said + chunk).slice(-2000))); + // A process that could not be started says so twice, as an error and as its end: it is told once. + let settled = false; + const done = (ok, why) => { + if (settled) return; + settled = true; + installs.delete(dir); + if (ok) log(`Installed its dependencies in ${Math.round((Date.now() - started) / 1000)} s`); + else log(`Could not install its dependencies: ${why}`); + resolve(ok); + }; + child.on("error", (error) => done(false, error.message)); + child.on("close", (code) => { + if (code !== 0) return done(false, `npm exited with ${code}${said.trim() ? `: ${said.trim().split("\n").slice(-3).join(" ")}` : ""}`); + const still = missingDependencies(dir); + done(!still.length, `still missing after npm: ${still.join(", ")}`); + }); + }); + installs.set(dir, install); + return install; +} diff --git a/extensions/office/fixtures/sample.docx b/extensions/office/fixtures/sample.docx new file mode 100644 index 000000000..348957848 Binary files /dev/null and b/extensions/office/fixtures/sample.docx differ diff --git a/extensions/office/fixtures/sample.xlsx b/extensions/office/fixtures/sample.xlsx new file mode 100644 index 000000000..4629c3c02 Binary files /dev/null and b/extensions/office/fixtures/sample.xlsx differ diff --git a/extensions/office/package-lock.json b/extensions/office/package-lock.json new file mode 100644 index 000000000..26c2710d2 --- /dev/null +++ b/extensions/office/package-lock.json @@ -0,0 +1,139 @@ +{ + "name": "pithagoras-office", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "pithagoras-office", + "version": "0.1.0", + "license": "Apache-2.0", + "dependencies": { + "jszip": "^3.10.1" + }, + "devDependencies": { + "react-icons": "5.7.0" + } + }, + "node_modules/core-util-is": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz", + "integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==", + "license": "MIT" + }, + "node_modules/immediate": { + "version": "3.0.6", + "resolved": "https://registry.npmjs.org/immediate/-/immediate-3.0.6.tgz", + "integrity": "sha512-XXOFtyqDjNDAQxVfYxuF7g9Il/IbWmmlQg2MYKOH8ExIT1qg6xc4zyS3HaEEATgs1btfzxq15ciUiY7gjSXRGQ==", + "license": "MIT" + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/isarray": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz", + "integrity": "sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ==", + "license": "MIT" + }, + "node_modules/jszip": { + "version": "3.10.2", + "resolved": "https://registry.npmjs.org/jszip/-/jszip-3.10.2.tgz", + "integrity": "sha512-3l+rb15IOWtUhU0H5MFqES/T6Kh7abYwjosBey/vD6hDt8zoEffkSC5Ws5SGtgVw3gBx2NEbhTeSW1+kWkpyTQ==", + "license": "(MIT OR GPL-3.0-or-later)", + "dependencies": { + "lie": "~3.3.0", + "pako": "~1.0.2", + "readable-stream": "~2.3.6", + "setimmediate": "^1.0.5" + } + }, + "node_modules/lie": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/lie/-/lie-3.3.0.tgz", + "integrity": "sha512-UaiMJzeWRlEujzAuw5LokY1L5ecNQYZKfmyZ9L7wDHb/p5etKaxXhohBcrw0EYby+G/NA52vRSN4N39dxHAIwQ==", + "license": "MIT", + "dependencies": { + "immediate": "~3.0.5" + } + }, + "node_modules/pako": { + "version": "1.0.11", + "resolved": "https://registry.npmjs.org/pako/-/pako-1.0.11.tgz", + "integrity": "sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw==", + "license": "(MIT AND Zlib)" + }, + "node_modules/process-nextick-args": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz", + "integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==", + "license": "MIT" + }, + "node_modules/react": { + "version": "19.3.1", + "resolved": "https://registry.npmjs.org/react/-/react-19.3.1.tgz", + "integrity": "sha512-QNhWdt7nrgVWWQG58QuGsAR3jv3Hbe4mb4vz7w2csnHavciZJgda1CPNaldF6eVxv90ud3jUXpKlHL69gO38yw==", + "dev": true, + "license": "MIT", + "peer": true, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/react-icons": { + "version": "5.7.0", + "resolved": "https://registry.npmjs.org/react-icons/-/react-icons-5.7.0.tgz", + "integrity": "sha512-LBLy340Rzqy6+/yVhZKT3B/QpP1BZaesGqasf09HPOBzRarcDIFH0WwXlXQfE7q7ipxK4MSiC5DIBWURCny6fw==", + "dev": true, + "license": "MIT", + "peerDependencies": { + "react": "*" + } + }, + "node_modules/readable-stream": { + "version": "2.3.8", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz", + "integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==", + "license": "MIT", + "dependencies": { + "core-util-is": "~1.0.0", + "inherits": "~2.0.3", + "isarray": "~1.0.0", + "process-nextick-args": "~2.0.0", + "safe-buffer": "~5.1.1", + "string_decoder": "~1.1.1", + "util-deprecate": "~1.0.1" + } + }, + "node_modules/safe-buffer": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", + "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==", + "license": "MIT" + }, + "node_modules/setimmediate": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/setimmediate/-/setimmediate-1.0.5.tgz", + "integrity": "sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA==", + "license": "MIT" + }, + "node_modules/string_decoder": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz", + "integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==", + "license": "MIT", + "dependencies": { + "safe-buffer": "~5.1.0" + } + }, + "node_modules/util-deprecate": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", + "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", + "license": "MIT" + } + } +} diff --git a/extensions/office/package.json b/extensions/office/package.json new file mode 100644 index 000000000..0416b1d08 --- /dev/null +++ b/extensions/office/package.json @@ -0,0 +1,22 @@ +{ + "name": "pithagoras-office", + "version": "0.1.0", + "description": "Office documents (docx, xlsx) edited in a full OnlyOffice editor that the portal runs in a container of its own: an Office tab for the library, tools so the agent can read and edit the same files, and a wake when a comment mentions it", + "type": "module", + "keywords": ["pi-package"], + "pi": {}, + "pithagoras": { + "client": "./web/client.js", + "style": "./web/client.css", + "server": "./server.js", + "requires": { "pithagoras-docker": "^1.1.0" } + }, + "scripts": { "build": "node build.mjs && tsc --noEmit -p ." }, + "dependencies": { + "jszip": "^3.10.1" + }, + "devDependencies": { + "react-icons": "5.7.0" + }, + "license": "Apache-2.0" +} diff --git a/extensions/office/scripts/make-fixtures.mjs b/extensions/office/scripts/make-fixtures.mjs new file mode 100644 index 000000000..fd22b5139 --- /dev/null +++ b/extensions/office/scripts/make-fixtures.mjs @@ -0,0 +1,205 @@ +/** + * Builds fixtures/sample.docx and fixtures/sample.xlsx, committed as binary alongside this script. + * + * Run manually (from the extension directory): + * + * env -u NODE_ENV node scripts/make-fixtures.mjs + * + * (NODE_ENV=production is set globally in this environment; the .ts import below rides Node's built-in + * type stripping, so plain `node` works -- no tsx needed. Re-run any time; the output is deterministic.) + * + * The parts are hand-written XML strings, but the ZIP bytes themselves come out of THIS extension's own + * writer (ooxml.writeZip): the fixtures therefore double as round-trip material for our zip reader and + * the ooxml parsers and editDocx, not just for any zip tool. The docx is deliberately richer than + * the blank template in store.ts + * template: multiple paragraphs (one empty), a sentence whose words sit in several w:t runs (the quote + * "revised after the" spans two of them -- test/fixtures.test.mts edits exactly that), entity-encoded + * characters (& < > — €), and word/comments.xml with two comments from different + * authors. The xlsx carries a sharedStrings part (including a multi-run <si> and an entity) AND inline + * strings across two sheets, with one sparse cell so the dense-grid fill is exercised too. + */ +import fs from "node:fs"; +import path from "node:path"; +import { writeZip } from "../src/ooxml.ts"; + +const HERE = import.meta.dirname; +const OUT = path.join(HERE, "..", "fixtures"); + +/* ------------------------------------------------------------------ */ +/* docx parts */ +/* ------------------------------------------------------------------ */ + +const DOCX_CONTENT_TYPES = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">` + + `<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>` + + `<Default Extension="xml" ContentType="application/xml"/>` + + `<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>` + + `<Override PartName="/word/comments.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.comments+xml"/>` + + `<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>` + + `</Types>`; + +const DOCX_ROOT_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>` + + `</Relationships>`; + +const W_NS = `xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"`; + +const DOCX_DOC_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml"/>` + + `<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/comments" Target="comments.xml"/>` + + `</Relationships>`; + +/** Five paragraphs: single-run, multi-run (the cross-run quote lives here), EMPTY, entity-laden, and a + * numeric-entity pair. The quotes below are load-bearing -- fixtures.test.mts asserts them verbatim and + * edits the cross-run one with ooxml.editDocx. */ +const DOCX_DOCUMENT = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<w:document ${W_NS}><w:body>` + + // p0 -- one run; named entities & and </> + `<w:p><w:r><w:t xml:space="preserve">Quarterly & annual report for <Meridian Labs></w:t></w:r></w:p>` + + // p1 -- THREE runs; "revised after the" crosses run 2 and run 3 (the fixture's cross-run quote) + `<w:p>` + + `<w:r><w:t xml:space="preserve">Revenue figures </w:t></w:r>` + + `<w:r><w:t>were revised after </w:t></w:r>` + + `<w:r><w:t>the audit found a discrepancy.</w:t></w:r>` + + `</w:p>` + + // p2 -- deliberately empty (parseDocx keeps it as "") + `<w:p/>` + + // p3 -- entities again, single run + `<w:p><w:r><w:t xml:space="preserve">Note: 5 & 7 were checked against the ledger (see <Appendix B>).</w:t></w:r></w:p>` + + // p4 -- numeric + hex character references (— = em dash, € = euro sign) + `<w:p>` + + `<w:r><w:t>Footnote: </w:t></w:r>` + + `<w:r><w:t>— total is €42,300.</w:t></w:r>` + + `</w:p>` + + `</w:body></w:document>`; + +const DOCX_STYLES = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><w:styles ${W_NS}></w:styles>`; + +/** Two comments, different authors: one dated multi-run (with an entity in its text), one undated single + * run carrying the @agent mention grammar ping.ts resolves (harmless in a fixture -- no wake fires offline). */ +const DOCX_COMMENTS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<w:comments ${W_NS}>` + + `<w:comment w:id="10" w:author="Alice Chen" w:date="2026-10-25T10:00:00Z">` + + `<w:p><w:r><w:t>Please fix </w:t></w:r><w:r><w:t xml:space="preserve">& resend the totals.</w:t></w:r></w:p>` + + `</w:comment>` + + `<w:comment w:id="25" w:author="Bob Ortiz">` + + `<w:p><w:r><w:t>@agent the appendix needs re-stating before we ship.</w:t></w:r></w:p>` + + `</w:comment>` + + `</w:comments>`; + +/* ------------------------------------------------------------------ */ +/* xlsx parts */ +/* ------------------------------------------------------------------ */ + +const XLSX_CONTENT_TYPES = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">` + + `<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>` + + `<Default Extension="xml" ContentType="application/xml"/>` + + `<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>` + + `<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>` + + `<Override PartName="/xl/worksheets/sheet2.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>` + + `<Override PartName="/xl/sharedStrings.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sharedStrings+xml"/>` + + `</Types>`; + +const XLSX_ROOT_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="xl/workbook.xml"/>` + + `</Relationships>`; + +const XLSX_WORKBOOK = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">` + + `<sheets><sheet name="Inventory" sheetId="1" r:id="rId1"/><sheet name="Notes" sheetId="2" r:id="rId2"/></sheets>` + + `</workbook>`; + +const XLSX_WORKBOOK_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet1.xml"/>` + + `<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet2.xml"/>` + + `</Relationships>`; + +/** Four shared strings: a plain one, an entity one (&), a MULTI-RUN <si> with a numeric reference + * (£ = pound sign -- run joining has to work here exactly like in docx runs), and a label. */ +const XLSX_SHARED_STRINGS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<sst xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" count="5" uniqueCount="4">` + + `<si><t>Product</t></si>` + + `<si><t>ACME & Sons Ltd</t></si>` + + `<si><r><t xml:space="preserve">Grand </t></r><r><t>total (£)</t></r></si>` + + `<si><t>Notes sheet</t></si>` + + `</sst>`; + +/** Sheet "Inventory": shared-string cells, a plain number, an INLINE string (with leading/trailing spaces + * to exercise xml:space), and a sparse hole -- row 3 has A, B and D but no C, so the grid must fill dense. */ +const XLSX_SHEET1 = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData>` + + `<row r="1">` + + `<c r="A1" t="s"><v>0</v></c>` + + `<c r="B1" t="s"><v>1</v></c>` + + `<c r="C1"><v>12</v></c>` + + `</row>` + + `<row r="3">` + + `<c r="A3" t="inlineStr"><is><t xml:space="preserve"> on order </t></is></c>` + + `<c r="B3" t="s"><v>2</v></c>` + + `<c r="D3"><v>99</v></c>` + + `</row>` + + `</sheetData></worksheet>`; + +/** Sheet "Notes": a shared string, a decimal number kept as stored text, and an inline string with padding. */ +const XLSX_SHEET2 = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData>` + + `<row r="1">` + + `<c r="A1" t="s"><v>3</v></c>` + + `<c r="B1"><v>1.5</v></c>` + + `</row>` + + `<row r="2">` + + `<c r="A2" t="inlineStr"><is><t xml:space="preserve"> see tab 1 </t></is></c>` + + `</row>` + + `</sheetData></worksheet>`; + +/* ------------------------------------------------------------------ */ +/* write */ +/* ------------------------------------------------------------------ */ + +function buildDocx() { + return writeZip([ + { name: "[Content_Types].xml", data: Buffer.from(DOCX_CONTENT_TYPES, "utf8") }, + { name: "_rels/.rels", data: Buffer.from(DOCX_ROOT_RELS, "utf8") }, + { name: "word/document.xml", data: Buffer.from(DOCX_DOCUMENT, "utf8") }, + { name: "word/_rels/document.xml.rels", data: Buffer.from(DOCX_DOC_RELS, "utf8") }, + { name: "word/styles.xml", data: Buffer.from(DOCX_STYLES, "utf8") }, + { name: "word/comments.xml", data: Buffer.from(DOCX_COMMENTS, "utf8") }, + ]); +} + +function buildXlsx() { + return writeZip([ + { name: "[Content_Types].xml", data: Buffer.from(XLSX_CONTENT_TYPES, "utf8") }, + { name: "_rels/.rels", data: Buffer.from(XLSX_ROOT_RELS, "utf8") }, + { name: "xl/workbook.xml", data: Buffer.from(XLSX_WORKBOOK, "utf8") }, + { name: "xl/_rels/workbook.xml.rels", data: Buffer.from(XLSX_WORKBOOK_RELS, "utf8") }, + { name: "xl/sharedStrings.xml", data: Buffer.from(XLSX_SHARED_STRINGS, "utf8") }, + { name: "xl/worksheets/sheet1.xml", data: Buffer.from(XLSX_SHEET1, "utf8") }, + { name: "xl/worksheets/sheet2.xml", data: Buffer.from(XLSX_SHEET2, "utf8") }, + ]); +} + +fs.mkdirSync(OUT, { recursive: true }); +const docx = buildDocx(); +const xlsx = buildXlsx(); +fs.writeFileSync(path.join(OUT, "sample.docx"), docx); +fs.writeFileSync(path.join(OUT, "sample.xlsx"), xlsx); +console.log(`fixtures/sample.docx ${docx.length} bytes (6 parts: document + comments + styles + rels)`); +console.log(`fixtures/sample.xlsx ${xlsx.length} bytes (7 parts: 2 sheets + sharedStrings + rels)`); diff --git a/extensions/office/src/callback.ts b/extensions/office/src/callback.ts new file mode 100644 index 000000000..0fa5c5c44 --- /dev/null +++ b/extensions/office/src/callback.ts @@ -0,0 +1,522 @@ +/** + * The OnlyOffice Document Server callback handler (plan 5.3): a plain node:http req/res function served by the + * loopback listener's POST /callback (server.ts, wave T-WIRE) -- NOT a portal route, because every portal route + * is session-gated and the DS container holds no cookie (plan 5.2 design decision). + * + * Flow (per plan 5.3/5.6): verify the HS256 JWT BEFORE dispatch, parse the envelope, branch on status exactly + * per the plan 5.3 table. The token arrives in the X-Auth-Token header -- that is where it lives because the + * container env we mint sets JWT_HEADER=X-Auth-Token and JWT_IN_BODY=false (src/container.ts dsEnv, plan 4); an + * embedded body.token is accepted ONLY as a defensive fallback for DS builds that ignore those vars. + * + * document.key format (FIXED by the contract): "office|" + docId + "|" + base64url-no-pad(sha256(String(expectedMtimeMs))). + * The hash pins the revision the config was minted against, so a stale session's save-back can never clobber a + * newer revision (plan 5.3: "the file's mtime/sha is the revision tools compare against"). The persist path + * re-derives the hash from the CURRENT store.getDoc(docId) meta and refuses on mismatch -- see persistSaved. + * + * Persistence (statuses 2/6 + the status-1 forcesave action): fetch body.url with retries (the DS can lag after + * a save), run store.persistBytes through its mtime gate, store the changesurl archive at + * dataDir/history/<docId>/<newMtimeMs>.zip (mkdir -p; best-effort -- the document write is what matters), fire + * every onPersisted(docId, buffer) hook (src/ping.ts registers its comment-mention wake through wirePing later), + * then apply the status's lock rule: 2 unlocks AFTER persist, 6 STAYS locked. Everything happens BEFORE the + * {"error":0} answer -- DS acts on the JSON error field, and any non-zero makes the editor show a save error / + * retry (plan 4). + * + * JWT for fetching body.url: plan 4 verifies the official example's plain fetch(req.body.url) and plan 5.3 just + * says "fetch url" -- the plan is SILENT on auth headers for that GET, though our image-wide env sets + * JWT_ENABLED=true. Belt-and-braces: we send a fresh 60 s HS256 token (claims {docId, key}) in X-Auth-Token -- + * harmless if DS's own download path is not gated, correct if it is. Documented choice, not a plan mandate. + * + * Wiring chosen here (the contract allows either): BOTH the contracted factory AND a bare module-level handler. + * - createCallbackHandler(getState, getSecret, opts?) returns an isolated handler -- tests build instances with + * their own tmp root/secret/retry budget and no global state at all. + * - configure({ getState, getSecret, root?, retryDelaysMs?, log? }) + export const handleCallback: T-WIRE's + * server.ts calls configure ONCE at boot with getState = () => container.readDsRecord() (ds.json is the single + * source of truth), getSecret = () => the jwt-secret string read once from dataDir()/jwt-secret, and the + * loopback listener dispatches `POST /callback -> callback.handleCallback(req, res)`. Before configure runs -- + * or whenever getState/getSecret answer null (container never ensured, no ds.json / no secret yet) -- every + * request gets 503 {"error":1} and nothing else happens. + */ +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import type { IncomingMessage, ServerResponse } from "node:http"; +import { dataDir, portal } from "./host"; +import { signJwt, verifyJwt } from "./jwt"; +import * as store from "./store"; +import * as live from "./live"; + +/** == the DsRecord shape of container.readDsRecord (kept local so this module does not import the Docker half). */ +export type CallbackState = { containerId: string; origin: string; imageTag: string }; + +export interface CallbackHandlerOptions { + /** Document-library root; production leaves it undefined (= host.dataDir()), unit tests pass a tmp dir. */ + root?: string; + /** Fixed backoff schedule BETWEEN attempts of the body.url / changesurl fetches. Default [100, 250] ms -> at + * most 3 attempts and ~350 ms of total waiting (plan 5.3: "retry fetch a few times (DS can lag)"; kept tiny on + * purpose so unit tests stay fast). */ + retryDelaysMs?: number[]; + /** Log sink for the status-3/7 lines and every failure; defaults to portal().log with a console.error fallback. */ + log?: (line: string) => void; +} + +type CallbackConfig = { getState: () => CallbackState | null; getSecret: () => string | null } & CallbackHandlerOptions; + +/** The factory the contract fixes: an isolated handler around these two lookups (+ optional test knobs). */ +export function createCallbackHandler( + getState: () => CallbackState | null, + getSecret: () => string | null, + opts?: CallbackHandlerOptions, +): (req: IncomingMessage, res: ServerResponse) => void { + return makeHandler({ getState, getSecret, ...opts }); +} + +/** Module-level configuration for the BARE handler below; T-WIRE's server.ts calls this exactly once at boot. */ +export function configure(config: CallbackConfig): void { + configured = config; +} + +let configured: CallbackConfig | null = null; + +/** The contracted bare handler (loopback listener POST /callback). 503 {"error":1} while unconfigured or until + * the DS record/secret exist. Fire-and-forget on purpose: all disk work completes before res.end, which is how + * callers (and tests) observe completion -- via the response. */ +export const handleCallback = (req: IncomingMessage, res: ServerResponse): void => { + if (!configured) { + respond(res, 503, 1); + return; + } + makeHandler(configured)(req, res); +}; + +/* ------------------------------------------------------------------ */ +/* document.key */ +/* ------------------------------------------------------------------ */ + +/** base64url-no-pad(sha256(String(expectedMtimeMs))) -- the revision hash embedded in every document.key. */ +export function revHashOf(mtimeMs: number): string { + return createHash("sha256").update(String(mtimeMs)).digest("base64url"); +} + +/** The only key shape this package mints (server.ts editor-config) and the only one parseKey accepts. */ +export function makeKey(docId: string, expectedMtimeMs: number): string { + return `office|${docId}|${revHashOf(expectedMtimeMs)}`; +} + +/** + * Parse "office|" + docId + "|" + base64url-no-pad(sha256(...)). Null when the prefix is missing, the docId part + * is empty, or the hash part is not 43 base64url characters (exactly one unpadded sha256 digest). The docId is + * taken greedily between the first and LAST pipe, so a future docId containing "|" still parses. + */ +export function parseKey(key: string): { docId: string; revHash: string } | null { + if (typeof key !== "string") return null; + const m = /^office\|(.+)\|[A-Za-z0-9_-]{43}$/.exec(key); + if (!m) return null; + const docId = m[1]; + if (docId === "") return null; + return { docId, revHash: key.slice(7 + docId.length + 1) }; +} + +/* ------------------------------------------------------------------ */ +/* persist hooks + the double-save guard */ +/* ------------------------------------------------------------------ */ + +const persistedHooks: Array<(docId: string, buffer: Buffer) => void> = []; + +/** Register a hook fired with (docId, EXACT saved bytes) after every successful persist -- src/ping.ts's + * comment-mention wake registers here through wirePing; the document save never waits on or dies with a hook. */ +export function onPersisted(fn: (docId: string, buffer: Buffer) => void): void { + persistedHooks.push(fn); +} + +/** + * Per-doc memo of our own last successful persist ({ url, mtimeMs }), module scope for the process's lifetime. + * WHY IT EXISTS: DS can report one force-save twice -- a status-1 envelope carries a forcesave action (type 2) + * whose result also arrives as the canonical status-6 callback. The second pass re-fetches with a key minted + * against the OLD mtime, so its revHash no longer matches the meta WE just wrote; without this memo that pass + * would look like an external agent edit and answer {error:1} (a spurious save-failure in the editor). The memo + * fires only when BOTH the document's current revision AND the download url equal ours -- a newer change at a + * different url still gets refused (plan 5.3: never overwrite a newer revision). Known accepted limit (report): + * typing between our forcesave and session close is dropped on that refusal -- exactly the stale-overwrite + * protection the plan demands; the user reloads via refreshFile (plan 5.7 tier 1). + */ +const lastPersist = new Map<string, { url: string; mtimeMs: number }>(); + +/* ------------------------------------------------------------------ */ +/* small plumbing */ +/* ------------------------------------------------------------------ */ + +/** Default log sink: portal().log once the server half set it, console.error before/without (tests stay quiet + * by passing their own). Same lazy pattern as src/container.ts. */ +function defaultLog(line: string): void { + try { + portal().log(`pithagoras-office: callback: ${line}`); + } catch { + console.error(`[pithagoras-office callback] ${line}`); + } +} + +function respond(res: ServerResponse, status: number, error: number): void { + if (res.headersSent) return; // already answered; an inner failure must not double-end + res.writeHead(status, { "Content-Type": "application/json" }); + res.end(JSON.stringify({ error })); +} + +/** The DS envelope is small (url + actions + users); a hard cap keeps one callback from eating memory. */ +const MAX_BODY_BYTES = 2 * 1024 * 1024; + +function readBody(req: IncomingMessage): Promise<Buffer> { + return new Promise((resolve, reject) => { + const chunks: Buffer[] = []; + let size = 0; + let failed = false; + req.on("data", (chunk: Buffer) => { + if (failed) return; + size += chunk.length; + if (size > MAX_BODY_BYTES) { + failed = true; + reject(Object.assign(new Error(`callback body exceeds ${MAX_BODY_BYTES} bytes`), { code: "body_too_large" })); + req.resume(); // drain without allocating + return; + } + chunks.push(chunk); + }); + req.on("end", () => resolve(Buffer.concat(chunks))); + req.on("error", reject); + }); +} + +function sleep(ms: number): Promise<void> { + return new Promise((r) => setTimeout(r, ms)); +} + +const DEFAULT_RETRY_DELAYS_MS = [100, 250]; + +/** GET with the documented JWT header and a fixed backoff schedule; returns the full body or throws the last + * error after 1 + delays.length attempts. HTTP non-2xx counts as a failure too (DS can answer a not-ready save). */ +async function fetchWithRetry(url: string, token: string, retryDelaysMs: number[]): Promise<Buffer> { + let attempt = 0; + for (;;) { + attempt += 1; + try { + const res = await fetch(url, { headers: { "X-Auth-Token": token } }); + if (!res.ok) throw new Error(`HTTP ${res.status}`); + return Buffer.from(await res.arrayBuffer()); + } catch (err) { + const delay = retryDelaysMs[attempt - 1]; + if (delay === undefined) throw err; + await sleep(delay); + } + } +} + +function writeAtomic(dest: string, bytes: Buffer): void { + fs.mkdirSync(path.dirname(dest), { recursive: true }); // mkdir -p history/<docId>/ + const tmp = path.join(path.dirname(dest), `.${path.basename(dest)}.${process.pid}.tmp`); + fs.writeFileSync(tmp, bytes); + fs.renameSync(tmp, dest); +} + +/* ------------------------------------------------------------------ */ +/* the status machine */ +/* ------------------------------------------------------------------ */ + +type Envelope = { + key?: unknown; + status?: unknown; + url?: unknown; + changesurl?: unknown; + users?: unknown; + actions?: unknown; + forcesavetype?: unknown; + token?: unknown; + [extra: string]: unknown; +}; + +function makeHandler(cfg: CallbackConfig): (req: IncomingMessage, res: ServerResponse) => void { + const log = cfg.log ?? defaultLog; + return (req, res) => { + void (async () => { + try { + await handleDispatch(log, cfg, req, res); + } catch (err) { + // Truly unexpected (fs failure in a path that should not throw, bug): one log line + a non-zero answer. + log(`internal callback error: ${(err as Error).message}`); + try { + respond(res, 500, 1); + } catch { + /* res already ended or broken pipe -- nothing more to do */ + } + } + })(); + }; +} + +async function handleDispatch( + log: (line: string) => void, + cfg: CallbackConfig, + req: IncomingMessage, + res: ServerResponse, +): Promise<void> { + // Route guard: the loopback listener serves exactly POST /callback (plan 5.2); reject anything else before + // reading a byte of body. The listener itself routes, so this only ever fires for misdirected probes. + if (req.method !== "POST") { + respond(res, 405, 1); + return; + } + const pathname = (req.url ?? "").split("?")[0]; + if (pathname !== "/callback" && pathname !== "/") { + respond(res, 404, 1); + return; + } + + let body: Envelope; + try { + const raw = await readBody(req); + body = JSON.parse(raw.toString("utf8")) as Envelope; + } catch (err) { + log(`malformed callback envelope: ${(err as Error).message}`); + respond(res, 400, 1); + return; + } + if (typeof body !== "object" || body === null || Array.isArray(body)) { + respond(res, 400, 1); + return; + } + + // DS configured at all? No ds record / no secret yet -> 503, nothing else. + const state = cfg.getState(); + const secret = cfg.getSecret(); + if (!state || !secret) { + respond(res, 503, 1); + return; + } + + // JWT BEFORE dispatch (plan 5.6): header X-Auth-Token (env pins JWT_HEADER=X-Auth-Token, JWT_IN_BODY=false), + // embedded body.token only as a defensive fallback. No token or any verification failure -> 401 {error:1}, + // and nothing below ever touches the disk. + const headerToken = req.headers["x-auth-token"]; + let token = typeof headerToken === "string" ? headerToken : Array.isArray(headerToken) ? headerToken[0] : undefined; + if (token === undefined && typeof body.token === "string") token = body.token; + // Verify the token ONCE and keep its claims: a second verifyJwt call below would add nothing except a race -- + // a token that expires in between makes that call return null and the envelope binding blow up as a 500. + if (token === undefined) { + log("rejected callback: missing or invalid JWT"); + respond(res, 401, 1); + return; + } + const verified = verifyJwt(token, secret); + if (verified === null) { + log("rejected callback: missing or invalid JWT"); + respond(res, 401, 1); + return; + } + + // Envelope binding (security): the envelope is taken from the SIGNED claims, never from the unsigned JSON + // body -- with a valid token alone anyone could otherwise POST any {key,status,url} and overwrite a library + // document. DS carries the callback envelope in the JWT payload (possibly wrapped under a 'payload' key); + // our own non-callback tokens (editor config, fetch) carry no envelope and are rejected below. The body is + // ignored from here on. + const claims = verified; // the verification made above -- exactly once per request + const signed = (claims.payload && typeof claims.payload === "object" && !Array.isArray(claims.payload)) + ? claims.payload as Record<string, unknown> + : claims; + if (typeof signed.key !== "string" || typeof signed.status !== "number" || !Number.isInteger(signed.status)) { + log("rejected callback: token does not carry a signed callback envelope (key + integer status)"); + respond(res, 400, 1); + return; + } + body = signed as Envelope; + + // Envelope shape. + const parsed = parseKey(body.key as string); // signed.key was verified a string above + if (!parsed) { + log("rejected callback: missing or malformed document.key"); + respond(res, 400, 1); + return; + } + const status = Number(body.status); + if (!Number.isInteger(status)) { + log(`rejected callback: non-integer status ${JSON.stringify(body.status)}`); + respond(res, 400, 1); + return; + } + + switch (status) { + case 1: { + // Being edited: set presence from body.users (plan 5.3: "lock = users.length>0") and apply the lock rule. + const snapshotIds = new Set<string>(); + if (Array.isArray(body.users)) { + for (const u of body.users) { + if (u && typeof u === "object" && typeof (u as Record<string, unknown>).id === "string") { + const id = (u as { id: string }).id; + const name = (u as { name?: unknown }).name; + live.setUser(parsed.docId, { id, name: typeof name === "string" ? name : undefined }, true); + snapshotIds.add(id); + } + } + } + for (const tracked of live.usersOf(parsed.docId)) if (!snapshotIds.has(tracked.id)) live.setUser(parsed.docId, { id: tracked.id }, false); + live.setLocked(parsed.docId, live.usersOf(parsed.docId).length > 0); // plan 5.3, status-1 row + + // Forcesave-click (actions[].type === 2) while the session continues -> persist NOW from body.url and STAY + // locked (status-6 rules; the canonical result may still arrive as a status-6 callback -- lastPersist + // absorbs that duplicate). The envelope's forcesavetype field belongs to statuses 6/7 only. + const actions = Array.isArray(body.actions) ? body.actions : []; + if (actions.some((a) => a && typeof a === "object" && (a as { type?: unknown }).type === 2) && typeof body.url === "string") { + await persistSaved(log, cfg, parsed.docId, parsed.revHash, body, false, res); // stays locked + return; + } + respond(res, 200, 0); + return; + } + + case 3: { + // Document saving error AT DS: log + audit only (audit.record is wired by T-WIRE; this log line IS the + // record until then). No disk write. We did not fail internally -> answer 0 so DS does not loop. + log(`status-3: document server reported a save error for ${parsed.docId}`); + respond(res, 200, 0); + return; + } + + case 7: { + // Force-saving error AT DS: same treatment as status 3 (one line each, no disk). + log(`status-7: document server reported a force-save error for ${parsed.docId} (forcesavetype=${JSON.stringify(body.forcesavetype ?? null)})`); + respond(res, 200, 0); + return; + } + + case 4: { + // Closed with no changes: terminal close -- nobody is connected anymore, so clear BOTH the flag and any + // tracked presence (no disk write of any kind). + live.clearDoc(parsed.docId); + respond(res, 200, 0); + return; + } + + case 2: + case 6: { + // Ready-for-saving / force-saved: persist (fetch url w/ retries -> persistBytes gate -> history zip -> + // hooks); status 2 unlocks AFTER the persist, status 6 STAYS locked (plan 5.3 rows). + await persistSaved(log, cfg, parsed.docId, parsed.revHash, body, status === 2, res); + return; + } + + default: { + log(`unknown callback status ${status} for ${parsed.docId}`); + respond(res, 400, 1); + return; + } + } +} + +/** + * The shared persist path (status 2, status 6 and the status-1 forcesave action). ALWAYS answers `res` itself: + * success -> 200 {"error":0}; every failure -> 200 {"error":1}. DS decides save outcomes from the JSON error + * field (plan 4: "must answer {\"error\":0}; anything else makes the editor show an error"), which is why the + * HTTP status stays 200 even for persist failures. + */ +async function persistSaved( + log: (line: string) => void, + cfg: CallbackConfig, + docId: string, + keyRevHash: string, + body: Envelope, + unlockAfter: boolean, + res: ServerResponse, +): Promise<void> { + const fail = (): void => respond(res, 200, 1); + const rootDir = () => cfg.root ?? dataDir(); + + const meta = store.getDoc(docId, rootDir()); + if (!meta) { + log(`persist for ${docId}: document no longer in the library -- refusing`); + return fail(); + } + + // STALE-GATE (plan 5.3): the key pins the revision the config was minted against; the CURRENT meta must match, + // else something already moved the revision and we must NOT overwrite it. The double-save memo below tells + // "we already wrote exactly this save" apart from "someone else won". + if (revHashOf(meta.mtimeMs) !== keyRevHash) { + const ours = lastPersist.get(docId); + const sameUrl = typeof body.url === "string" && ours?.url === body.url; + if (ours && ours.mtimeMs === meta.mtimeMs && sameUrl) { + // The duplicate pass of a force-save we already persisted: report SUCCESS, apply the status's lock rule. + log(`persist for ${docId}: this save was already persisted at revision ${meta.mtimeMs} -- treating as done`); + if (unlockAfter) live.clearDoc(docId); // a status-2 re-send closes the session + else live.setLocked(docId, true); // status 6: editing continues, stay locked + respond(res, 200, 0); + return; + } else { + log(`persist for ${docId}: stale key (current revision ${meta.mtimeMs}, expected hash ${keyRevHash}) -- NOT overwriting a newer revision`); + if (unlockAfter) live.clearDoc(docId); // terminal close: the session is over, drop flag AND presence + } + return fail(); + } + + if (typeof body.url !== "string" || body.url === "") { + log(`persist for ${docId}: callback carries no url to download`); + return fail(); + } + + // JWT for the outbound GET (documented choice -- plan silent, see module header): a fresh 60 s HS256 token, + // purpose-tagged "fetch" (S3) so it can never masquerade as an editor-config or a callback in any verifier. + const fetchToken = signJwt({ docId, key: body.key }, 60_000, secretOrThrow(cfg), "fetch"); + let bytes: Buffer; + try { + bytes = await fetchWithRetry(body.url, fetchToken, cfg.retryDelaysMs ?? DEFAULT_RETRY_DELAYS_MS); + } catch (err) { + log(`persist for ${docId}: could not download ${body.url} after ${(cfg.retryDelaysMs ?? DEFAULT_RETRY_DELAYS_MS).length + 1} attempts: ${(err as Error).message}`); + // Leave the lock state untouched -- DS will retry; a status-2 close without a save stays visible. + return fail(); + } + + let newMtimeMs: number; + try { + newMtimeMs = store.persistBytes(meta, bytes, rootDir()).mtimeMs; + } catch (err) { + if ((err as { code?: string }).code === "revision_mismatch") { + log(`persist for ${docId}: revision moved between read and write -- NOT overwriting`); + if (unlockAfter) live.clearDoc(docId); // terminal close, same rule as the stale gate above + } else { + log(`persist for ${docId}: store failure: ${(err as Error).message}`); + } + return fail(); + } + + lastPersist.set(docId, { url: body.url, mtimeMs: newMtimeMs }); + + // Changes archive -> dataDir/history/<docId>/<newRevision>.zip (rev = the NEW post-persist mtimeMs; the index + // carries that number now too, so history entries line up with what office_edit and tools gate against). + // Best-effort: a lost archive costs the editor's version-history view, never the document. + if (typeof body.changesurl === "string" && body.changesurl !== "") { + try { + const zip = await fetchWithRetry(body.changesurl, fetchToken, cfg.retryDelaysMs ?? DEFAULT_RETRY_DELAYS_MS); + writeAtomic(path.join(rootDir(), "history", docId, `${newMtimeMs}.zip`), zip); + } catch (err) { + log(`persist for ${docId}: could not store the changes archive: ${(err as Error).message}`); + } + } + + // Wake hooks (ping.ts comment mentions register here): exact saved bytes; one bad hook must not kill the rest. + for (const hook of persistedHooks) { + try { + hook(docId, bytes); + } catch (err) { + log(`onPersisted hook failed for ${docId}: ${(err as Error).message}`); + } + } + + if (unlockAfter) live.clearDoc(docId); // status 2: terminal close AFTER persist -- drop flag AND presence + else live.setLocked(docId, true); // status 6: editing continues, stay locked + + // Success is only answered once the document AND the lock state are settled -- DS reads this JSON field to + // decide the save outcome (plan 4). Every earlier branch already answered itself. + respond(res, 200, 0); +} + +/** The dispatch already verified a non-null secret; keep persistSaved's signature lean. */ +function secretOrThrow(cfg: CallbackConfig): string { + const secret = cfg.getSecret(); + if (!secret) throw Object.assign(new Error("jwt secret vanished mid-dispatch"), { code: "no_secret" }); + return secret; +} diff --git a/extensions/office/src/client.tsx b/extensions/office/src/client.tsx new file mode 100644 index 000000000..4866c72f7 --- /dev/null +++ b/extensions/office/src/client.tsx @@ -0,0 +1,242 @@ +import "../style.css"; +import { Suspense, useState } from "react"; +import type { ChatProps, PanelProps, Portal, ToolCallView } from "@pithagoras/extension-api"; +import { LuFileText, LuPresentation, LuTable } from "react-icons/lu"; +import type { ReactNode } from "react"; +import { officeUrl, portal, setPortal, t } from "./portal"; +import { refreshOfficeState, useOfficeState } from "./web/state"; +import { api, sayError } from "./web/api"; +import { compareDocs, loadSort, saveSort, SortBar, type SortChoice } from "./web/sort"; +import { editorsIn } from "./office-ui"; + +/** + * pithagoras-office: its browser half. The Office tab -- the full library with its own workbench chrome (style.css), + * opened documents in a full-bleed OnlyOffice editor; the office panel beside an agent/project's home chat, the tool + * cards of its office_* calls, and its page in Settings where the Document Server is run. The docs extension's slot + * shape is mirrored: nav.page + chat.panel + chat.tool entries + settings.page (docs/web/client.js). Its heavy parts + * load as chunks, the way Understory's do (ui.lazy + Suspense). + */ + +/** A document beside a home chat: every UNOWNED one (workspace-scanned or human-made -- shared by all chats), + * plus ones it owns or a comment mention granted it -- derived from GET /state (the server half keeps no per-chat + * route; owner and grants ride in every doc's meta, plan 2.1). The sort choice is the one of web/sort.tsx, + * persisted under "po-sort" so this panel and the full Office page share it. */ +function OfficePanel({ chat }: PanelProps) { + portal.locale.use(); + const state = useOfficeState(); + const [sort, setSort] = useState<SortChoice>(loadSort); + if (!state) return <portal.ui.RowsSkeleton rows={3} label={t("Loading…")} />; + const chooseSort = (next: SortChoice) => { + setSort(next); + saveSort(next); + }; + const mine = [...state.docs] + .filter((d) => d.owner === null || d.owner === chat.id || d.grants.some((g) => g.chatId === chat.id)) + .sort((a, b) => compareDocs(a, b, sort.key, sort.dir)); + return ( + <div className="space-y-1 p-2"> + <PanelNewDocument /> + {mine.length === 0 ? ( + <portal.ui.Empty>{t("No office documents yet -- the workspace .docx and .xlsx files, human-made docs, and the ones this chat owns or was granted by a comment mention will appear here.")}</portal.ui.Empty> + ) : ( + <> + <SortBar sort={sort} onSort={chooseSort} /> + {mine.map((doc) => { + const n = editorsIn(state.locks, doc.docId); + return ( + <div key={doc.docId} className="po-row"> + <button type="button" className="po-open" onClick={() => portal.navigate(officeUrl(doc.docId))}> + {doc.ext === "docx" ? ( + <LuFileText className="po-ext po-ext-word" aria-hidden /> + ) : ( + <LuTable className="po-ext po-ext-cell" aria-hidden /> + )} + <span style={{ minWidth: 0 }}> + <span className="po-title">{doc.title}</span> + <span className="po-meta"> + {t("{title}.{ext}", { title: doc.title, ext: doc.ext })} + {n > 0 ? ` · ${n === 1 ? t("1 editing now") : t("{n} editing now", { n })}` : ""} + </span> + </span> + </button> + </div> + ); + })} + </> + )} + </div> + ); +} + +/** The panel's slim new-document row: title + a word/sheet kind toggle + create, on the SAME api.createDocument + * as the page's NewDocument -- and like it it passes NO owner, so a doc made here is unowned and lands where the + * workspace docs already are: every chat's panel and the global library. */ +function PanelNewDocument() { + portal.locale.use(); + const [title, setTitle] = useState(""); + const [kind, setKind] = useState<"word" | "cell">("word"); + const [busy, setBusy] = useState(false); + const [problem, setProblem] = useState<string | null>(null); + const make = async () => { + if (title.trim() === "" || busy) return; + setBusy(true); + setProblem(null); + try { + await api.createDocument({ title: title.trim(), type: kind }); + setTitle(""); + void refreshOfficeState(); // the new doc appears in this panel immediately, not on the next 5 s poll + } catch (e) { + setProblem(sayError(e)); + } finally { + setBusy(false); + } + }; + return ( + <form + className="po-newcompact" + onSubmit={(e) => { + e.preventDefault(); + void make(); + }} + > + <input + className={portal.ui.classes.input} + value={title} + maxLength={200} + placeholder={t("New document title")} + aria-label={t("New document title")} + onChange={(e) => setTitle(e.target.value)} + /> + <span className="po-kind"> + <button type="button" aria-pressed={kind === "word"} className={`po-kind-btn${kind === "word" ? " po-kind-on" : ""}`} onClick={() => setKind("word")} title={t("Word (.docx)")} aria-label={t("Word (.docx)")}> + <LuFileText aria-hidden /> + </button> + <button type="button" aria-pressed={kind === "cell"} className={`po-kind-btn${kind === "cell" ? " po-kind-on" : ""}`} onClick={() => setKind("cell")} title={t("Sheet (.xlsx)")} aria-label={t("Sheet (.xlsx)")}> + <LuTable aria-hidden /> + </button> + </span> + <button type="submit" className={portal.ui.classes.primarySmall} disabled={busy || title.trim() === ""}> + {t("Add")} + </button> + {problem && ( + <portal.ui.ErrorBanner onClose={() => setProblem(null)}>{t("The document could not be made: {error}", { error: problem })}</portal.ui.ErrorBanner> + )} + </form> + ); +} + +/** What the office tools reported, beside their text: which document, and a way to open it. The portal's own card + * says it best while a call runs or failed, so those stay as they are (the docs extension's tool-card precedent). */ +interface OfficeToolDetails { + action?: string; + id?: string; + title?: string; + ext?: "docx" | "xlsx"; + mtimeMs?: number; + from?: number; + to?: number; + ops?: number; + documents?: unknown[]; +} + +function OfficeCall(props: ChatProps & { call: ToolCallView; card: ReactNode }) { + portal.locale.use(); + const state = useOfficeState(); + const { call, card } = props; + const d = call.details as OfficeToolDetails | undefined; + if (call.status !== "done" || !d || typeof d.action !== "string") return card; + + const heading: Record<string, string> = { + create: t("Created a document"), + read: t("Read a document"), + edit: t("Edited a document"), + comments: t("Read the comments of a document"), + list: t("Listed the office documents"), + }; + const titleText = heading[d.action]; + if (!titleText) return card; + + // The tool names its document by id alone; its title comes from what the poll has (or the call said it). + let name: string | null = d.title ?? null; + if (d.id) name ??= state?.docs.find((x) => x.docId === d.id)?.title ?? null; + const facts: string[] = []; + if (d.action === "list") { + facts.push(d.documents?.length === 1 ? t("1 document") : t("{n} documents", { n: d.documents?.length ?? 0 })); + } else if (d.id) { + const ext = d.ext ?? state?.docs.find((x) => x.docId === d.id)?.ext; + facts.push(ext ? `${name ?? d.id}.${ext}` : name ?? d.id); + if (d.action === "edit") { + facts.push(typeof d.from === "number" && typeof d.to === "number" ? t("Revision {from} to {to}", { from: d.from, to: d.to }) : typeof d.to === "number" ? t("Revision {n}", { n: d.to }) : ""); + facts.push(typeof d.ops === "number" ? (d.ops === 1 ? t("1 edit") : t("{n} edits", { n: d.ops })) : ""); + } else if (typeof d.mtimeMs === "number") facts.push(t("Revision {n}", { n: d.mtimeMs })); + } + + const open = d.action === "list" ? officeUrl() : typeof d.id === "string" && d.id !== "" ? officeUrl(d.id) : null; + if (!open) return card; // nothing with an address to open: the portal's card already said enough + + return ( + <div className="flex items-center gap-2 rounded-lg border border-line bg-raised/40 p-2"> + <LuPresentation /> + <div style={{ minWidth: 0, flex: 1 }}> + <span className="text-xs text-fg-muted">{titleText}</span> + {name && <p className="truncate text-sm text-fg">{name}</p>} + {facts.filter(Boolean).length > 0 && <p className="text-[11px] text-fg-subtle">{facts.filter(Boolean).join(" · ")}</p>} + </div> + <button type="button" className={portal.ui.classes.ghost} onClick={() => portal.navigate(open)}> + {t("Open in Office")} + </button> + </div> + ); +} + +export default function office(given: Portal): void { + setPortal(given); + // Its heavy parts, fetched only when they are opened (the Understory idiom). + const Page = portal.ui.lazy(() => import("./web/library"), "OfficePage"); + const Settings = portal.ui.lazy(() => import("./web/settings"), "OfficeSettings"); + const fallback = <div className="flex flex-1 items-center justify-center text-sm text-fg-subtle">{t("Loading…")}</div>; + + // The Office tab: the full library at /x/pithagoras-office/office, its chrome its own (style.css), a document + // open as ?doc=<id> in the whole page. + portal.slots.register("nav.page", { + id: "office", + title: t("Office"), + icon: LuPresentation, + component: () => ( + <Suspense fallback={fallback}> + <Page /> + </Suspense> + ), + }); + + // Beside every chat -- shown for all, as the docs' panel is: every unowned (workspace-scanned / human-made) doc + // is shared by every chat, plus what this one owns or was granted; where there is nothing at all, it says so. + portal.slots.register("chat.panel", { + id: "office", + name: "office", + title: t("Office"), + icon: LuPresentation, + component: OfficePanel, + useButton: () => ({ title: t("The office documents this chat owns or was granted") }), + }); + + // A call of one of the tools, in place of the portal's card for a finished one that names its document. + for (const tool of ["office_list", "office_create", "office_read", "office_edit", "office_comments"]) { + portal.slots.register("chat.tool", { id: tool, tool, component: OfficeCall }); + } + + // Its page in Settings: the Document Server's service panel. + portal.slots.register("settings.page", { + id: "office", + title: t("Office"), + icon: LuPresentation, + search: [ + { title: t("Office"), words: "office document docx xlsx word sheet spreadsheet onlyoffice document server container image tag pull co-editing library workspace scan rescan export trash" }, + ], + component: () => ( + <Suspense fallback={<p className="text-sm text-fg-subtle">{t("Loading…")}</p>}> + <Settings /> + </Suspense> + ), + }); +} diff --git a/extensions/office/src/container.ts b/extensions/office/src/container.ts new file mode 100644 index 000000000..da7c645ec --- /dev/null +++ b/extensions/office/src/container.ts @@ -0,0 +1,336 @@ +/** + * pithagoras-office container lifecycle: the OnlyOffice Document Server container behind every edit. + * + * The container is plain Docker (pithagoras-docker) with: + * - a pinned image tag from settings (never :latest -- plan 5.1), + * - the portal's shared network (`HostConfig.NetworkMode = await docker.sharedNetwork()`), so the DS container + * and the in-process loopback listener reach each other on 127.0.0.1 in BOTH deployment shapes (plan 5.1/5.2), + * - a named volume for /var/www/onlyoffice/Data (DS's internal storage; plan 5.1), + * - exactly the four JWT env vars the 10.x community image documents (plan 4): one secret covers config tokens + * AND callback verification in every direction, so a per-direction setup is unnecessary. + * + * The DS origin -- where the editor page and document.url point -- is fixed by the deployment shape (there are no + * published ports under shared networking; the stock nginx listens on :80 of the whole shared netns) and is + * recorded in ds.json at create time, which stays the single source of truth for it: + * - host mode -> "http://127.0.0.1" (DS sits on the machine's own loopback). + * - container:<id> mode -> the recorded origin is the string `container:<id>` itself -- NOT a URL yet. The plan + * defers browser reach in that shape to the M0 spike-2 decision (nginx bind override vs portal reverse-proxy vs + * declared local-only), so there is no honest browser-facing URL to write at create time; this marker carries the + * portal container id so a later wave can turn it into one. DS-side fetches (document.url, callback) use the + * loopback listener's own address and never read this field. + * + * What the plan does NOT mandate here: a HostConfig.Memory cap. The ~4 GB number in the plan is a RAM RECOMMENDATION + * ("system requirements RAM >= 4 GB", surfaced as a warning before first start), not an enforced limit -- no cap is + * set, so a document save under load cannot be OOM-killed mid-write. (Documented divergence.) + * + * Container identity/adopt: `containers.get(DS_NAME)` finds one the extension made; if it is there we NEVER recreate + * or restart it -- a running one yields { started: false }, a stopped one is simply started. A container the portal + * ran before this extension existed (ownerless) is still usable through the same name; adopting it into this + * extension's ownership is left to the Docker Settings page (out of scope for the lifecycle here). + * + * User intent: `stopContainer` records a user-stopped marker FILE in the data dir (`user-stopped`) -- a file, not + * memory, so a portal restart does not revive what the user stopped (plan 5.1: "lookAfter keeps a user-started + * container alive but never resurrects one the user stopped"). Any `ensureContainer` success clears it: an explicit + * ensure is by definition a user start request (settings enable / POST /container/start), so self-clearing keeps the + * two functions consistent without a second "unstop" call. + * + * Module state: `pullState` (the last pull progress reported to status()) is plain module scope on purpose. The + * server half outlives re-imports within one process, and the in-memory pull position is ephemeral by contract + * ("held in memory", plan 5.1) -- a restart simply starts with no progress to show until the next ensure. + */ +import fs from "node:fs"; +import path from "node:path"; +import { dataDir, docker as hostDocker, portal } from "./host"; +import { loadSettings } from "./settings"; +import type { ContainerInfo, Docker, PullState } from "../../docker/src/api"; + +/** The DS container, by that name: the adopt key of `containers.get` and every lifecycle call. */ +export const DS_NAME = "pithagoras-office-ds"; +/** The named volume bound at /var/www/onlyoffice/Data (DS's internal storage). */ +export const DS_VOLUME = "pithagoras-office-ds-data"; +const DS_DATA_PATH = "/var/www/onlyoffice/Data"; +/** Where the version of a running server is read from inside the container. */ +const GIT_VERSION_FILE = "/var/www/onlyoffice/documentserver/GitVersion.json"; + +/** The marker file that records "the user stopped it": lookAfter must not revive it. Lives in the data dir so the + * intent survives a portal restart. */ +export const USER_STOPPED_FILE = "user-stopped"; + +type DsRecord = { containerId: string; origin: string; imageTag: string }; + +/** The last pull progress, in memory only, so status() can report it without re-pulling (contract + plan 5.1). */ +let pullState: PullState | null = null; + +/** One log line: portal().log where the server half started it, console.error elsewhere -- lazy, so unit tests that + * pass explicit docker/root never touch portal() at all. */ +function log(line: string): void { + try { + portal().log(`pithagoras-office: ${line}`); + } catch { + console.error(`[pithagoras-office] ${line}`); + } +} + +function noDocker(): Error { + return Object.assign(new Error("no_docker"), { code: "no_docker" }); +} + +/** docker ?? host.docker(), guarded: undefined or unreachable -> the no_docker error, as the contract fixes it. */ +async function reachable(docker?: Docker): Promise<Docker> { + const d = docker ?? hostDocker(); + if (!d || !(await d.available())) throw noDocker(); + return d; +} + +function resolvedRoot(root?: string): string { + return root ?? dataDir(); +} + +/** The exact JWT env of the DS container (plan 4: four vars, one secret, covering every direction). The secret is + * the trimmed content of dataDir()/jwt-secret, read HERE independently (contract amendment; server.ts creates the + * file at boot, so a missing file surfaces as its plain fs error). */ +function dsEnv(root: string): string[] { + const secret = fs.readFileSync(path.join(root, "jwt-secret"), "utf8").trim(); + return [ + "JWT_ENABLED=true", + `JWT_SECRET=${secret}`, + "JWT_HEADER=X-Auth-Token", + "JWT_IN_BODY=false", + ]; +} + +function writeAtomic(dest: string, bytes: Buffer): void { + const dir = path.dirname(dest); + const tmp = path.join(dir, `.${path.basename(dest)}.${process.pid}.tmp`); + fs.writeFileSync(tmp, bytes); + fs.renameSync(tmp, dest); +} + +/** ds.json -- the single source of truth for container id and origin (written on every ensure). */ +export function readDsRecord(root?: string): DsRecord | null { + let raw: unknown; + try { + raw = JSON.parse(fs.readFileSync(path.join(resolvedRoot(root), "ds.json"), "utf8")); + } catch { + return null; + } + if (typeof raw !== "object" || raw === null) return null; + const r = raw as Record<string, unknown>; + if (typeof r.containerId !== "string" || typeof r.origin !== "string" || typeof r.imageTag !== "string") return null; + return { containerId: r.containerId, origin: r.origin, imageTag: r.imageTag }; +} + +function dsRecordPath(root: string): string { + return path.join(root, "ds.json"); +} + +/** Whether the user-stopped marker is present. */ +export function userStopped(root?: string): boolean { + try { + fs.accessSync(path.join(resolvedRoot(root), USER_STOPPED_FILE)); + return true; + } catch { + return false; + } +} + +function clearUserStopped(root: string): void { + fs.rmSync(path.join(root, USER_STOPPED_FILE), { force: true }); +} + +/** The named volume for /var/www/onlyoffice/Data, created once: check `volumes.list()` first and create when absent. + * A concurrent creation surfaces as DockerError status 409 ("already exists") -- or the API keeps the existing own + * volume by that name outright (its documented behavior) -- both are swallowed, anything else propagates. */ +async function ensureVolume(d: Docker): Promise<void> { + const listed = await d.volumes.list(); + if (listed.some((v) => v.name === DS_VOLUME)) return; + try { + await d.volumes.create(DS_VOLUME); + } catch (e) { + const err = e as { status?: number; message?: string }; + const alreadyExists = err.status === 409 || /already exists/i.test(err.message ?? ""); + if (!alreadyExists) throw e; + } +} + +/** + * Image ensured, container present and running: `{ started: false }` where one already ran (never recreated or + * restarted), else it is made -- volume first, then `containers.create(DS_NAME, spec)` under the exact name (the + * wrapper takes the name as its own parameter, not a field of the Docker create body) -- and started. ds.json is + * rewritten on every success so it always reflects this container and the current deployment shape. + */ +// Concurrency guard: serialize the WHOLE ensure so two simultaneous callers (portal start + a status poller) +// cannot race create/start/ds.json. Cleared in finally so a failed run never poisons the next attempt. +let ensuring: Promise<{ started: boolean }> | null = null; + +export function ensureContainer( + docker?: Docker, + onPull?: (state: PullState) => void, + root?: string, +): Promise<{ started: boolean }> { + ensuring ??= ensureContainerOnce(docker, onPull, root).finally(() => { + ensuring = null; + }); + return ensuring; +} + +async function ensureContainerOnce( + docker?: Docker, + onPull?: (state: PullState) => void, + root?: string, +): Promise<{ started: boolean }> { + const d = await reachable(docker); + const r = resolvedRoot(root); + const tag = loadSettings(r).imageTag; + + // One pull for the image at a time, whoever asks; progress held in memory for status() AND handed to onPull. + await d.images.ensure(tag, (state) => { + pullState = state; + onPull?.(state); + }); + + // The deployment shape decides both the NetworkMode and the recorded origin (host: "http://127.0.0.1", + // container:<id>: that string itself, turned into a real URL by whatever M0 spike-2 picks). + const networkMode = await d.sharedNetwork(); + + let containerId: string; + let existing = await d.containers.get(DS_NAME); + // A changed imageTag applies at the next start: a stopped (or created-but-never-started) container left from an + // older tag is removed so the creation path below rebuilds it with the CURRENT image. A running container is + // never touched -- no surprise restart; it picks up the new image at its next stop. + if (existing && existing.state !== "running" && existing.image !== tag) { + await d.containers.remove(DS_NAME); + existing = undefined; // let the creation path below recreate it with the new image + } + if (!existing) { + await ensureVolume(d); + const spec = { + Image: tag, + Env: dsEnv(r), + HostConfig: { + NetworkMode: networkMode, + Binds: [`${DS_VOLUME}:${DS_DATA_PATH}`], + }, + }; + containerId = (await d.containers.create(DS_NAME, spec)).id; + } else { + // Adopted as-is by name: no recreate, no restart. Its volume is already bound at that path. + containerId = existing.id; + } + + const started = !existing || existing.state !== "running"; + if (started) await d.containers.start(DS_NAME); + + writeAtomic(dsRecordPath(r), Buffer.from(JSON.stringify({ containerId, origin: networkMode === "host" ? "http://127.0.0.1" : networkMode, imageTag: tag }, null, 2) + "\n")); + // An explicit ensure is a start request from the user: whatever stopped it before, keep it running from now on. + clearUserStopped(r); + return { started }; +} + +/** Stops the container by name and records the user-stopped intent (marker file, BEFORE the stop, so a lookAfter in + * flight cannot race the user back into a revive). A missing or already-exited container needs no stop; the intent + * stays recorded -- the user asked for "kept stopped", which is exactly what lookAfter honors. */ +export async function stopContainer(docker?: Docker, root?: string): Promise<void> { + const d = await reachable(docker); + const r = resolvedRoot(root); + fs.mkdirSync(r, { recursive: true }); + writeAtomic(path.join(r, USER_STOPPED_FILE), Buffer.from("user stopped the document server\n")); + const existing = await d.containers.get(DS_NAME); + if (existing && existing.state !== "exited") await d.containers.stop(DS_NAME); +} + +/** + * The running version: exactly `cat /var/www/onlyoffice/documentserver/GitVersion.json` in the container -- the + * build file the stock image ships, whose DocEditorVersion is what DS reports about itself (the plan only says + * "exec into container"; M0 re-verifies the path on the pinned tag). Never an error: a probe failure just leaves + * `version` absent. + */ +async function probeVersion(d: Docker): Promise<string | undefined> { + try { + const res = await d.containers.exec(DS_NAME, ["cat", GIT_VERSION_FILE], { timeoutMs: 10_000 }); + if (res.exitCode !== 0) return undefined; + const body = JSON.parse(res.output.replace(/^\uFEFF/, "").trim()) as Record<string, unknown>; + const version = body.DocEditorVersion ?? body.Version; + return typeof version === "string" && version ? version : undefined; + } catch { + return undefined; + } +} + +/** + * Container state for the status block: absent or exited/created -> "stopped"; otherwise Docker's own inspect word + * (with `State.Starting` folded in as "starting"). The current memory usage is read from the same inspect body's + * top-level `Memory` where present. Pull progress comes from the in-memory last state of an ensure. + */ +export async function status( + docker?: Docker, + root?: string, +): Promise<{ state: string; version?: string; memBytes?: number; pull?: PullState | null }> { + const out: { state: string; version?: string; memBytes?: number; pull?: PullState | null } = { + state: "stopped", + pull: pullState ? { ...pullState } : null, + }; + let d: Docker | undefined; + try { + d = await reachable(docker); + } catch { + return out; // no docker here: the container is as good as stopped from our seat + } + + let existing: ContainerInfo | undefined; + try { + existing = await d.containers.get(DS_NAME); + } catch { + existing = undefined; + } + if (!existing || existing.state === "exited" || existing.state === "created") return out; + + let inspected: Record<string, unknown> | undefined; + try { + inspected = await d.containers.inspect(DS_NAME); + } catch { + inspected = undefined; + } + const st = (inspected?.State ?? null) as { Status?: string; Starting?: boolean } | null; + if (!st || typeof st.Status !== "string") { + // The list view said something and inspect is silent: pass the list word through. + out.state = existing.state; + return out; + } + out.state = st.Status === "running" && st.Starting ? "starting" : st.Status === "exited" ? "stopped" : st.Status; + + const mem = inspected?.Memory; + if (typeof mem === "number" && mem > 0) out.memBytes = mem; + if (out.state === "running") { + const version = await probeVersion(d); + if (version) out.version = version; + } + return out; +} + +/** + * Fire-and-forget: settings enabled, user-stopped intent clear, container not running -> ensureContainer. Every + * failure lands as one log line (never a throw -- it is called from the boot settle and the 30 s unref'd interval). + */ +export function lookAfter(docker?: Docker, root?: string): void { + void (async () => { + let d: Docker; + let r: string; + try { + d = await reachable(docker); + r = resolvedRoot(root); + } catch (e) { + log(`lookAfter skipped: ${(e as Error).message}`); + return; + } + try { + if (!loadSettings(r).enabled) return; + if (userStopped(r)) return; + const existing = await d.containers.get(DS_NAME); + if (existing?.state === "running") return; + await ensureContainer(d, undefined, r); + } catch (e) { + log(`could not look after the document server container: ${(e as Error).message}`); + } + })(); +} diff --git a/extensions/office/src/host.ts b/extensions/office/src/host.ts new file mode 100644 index 000000000..caafdeeb2 --- /dev/null +++ b/extensions/office/src/host.ts @@ -0,0 +1,26 @@ +import type { ServerPortal } from "@pithagoras/extension-api/server"; +import type { Docker } from "../../docker/src/api"; + +/** + * The portal this server half was started with, for every module of it to reach: Docker, the data folder, the log. + * Set as the half starts (see server.ts), before anything else of it runs, and set again by a start after a stop: the + * modules are loaded once, and outlive a stop. + */ +let current: ServerPortal | undefined; + +export function setPortal(portal: ServerPortal): void { + current = portal; +} + +export function portal(): ServerPortal { + if (!current) throw new Error("pithagoras-office's server half has not started"); + return current; +} + +/** This extension's own data folder (settings, jwt-secret, the document library). */ +export const dataDir = (): string => portal().dataDir(); + +/** Docker, which pithagoras-office requires (`pithagoras.requires` in its package.json): there whenever this half runs. */ +export function docker(): Docker | undefined { + return portal().extensions.use("pithagoras-docker"); +} diff --git a/extensions/office/src/jszip.ts b/extensions/office/src/jszip.ts new file mode 100644 index 000000000..92fa64598 --- /dev/null +++ b/extensions/office/src/jszip.ts @@ -0,0 +1,38 @@ +/** + * The jszip loader: NO module of this package imports jszip statically. The built server.js is imported by the + * portal BEFORE the extension's own node_modules are guaranteed to exist (a hand-copied folder has none), and + * deps.ensureDependencies -- wired into the server entry at boot -- installs them in the BACKGROUND, exactly as + * pithagoras-docs does for better-sqlite3. So the first template/import call here AWAITs the armed install promise + * and only then imports jszip; the synchronous parts of the store never touch it. A load that fails is not cached: + * the next call retries after a later boot re-arms or the install settles. + */ +import type JSZip from "jszip"; + +/** The loaded package's shape: the constructor AND its statics (JSZip.loadAsync). */ +export type JSZipStatic = typeof JSZip; + +let install: Promise<boolean> | null = null; +let load: Promise<JSZipStatic> | null = null; + +/** Arm this loader with the boot-time install (the server entry wires deps.ensureDependencies through here). + * The first jsZip() call waits on it before importing. */ +export function armJsZipInstall(promise: Promise<boolean>): void { + install = promise; +} + +/** jszip, at most one load per boot. `typeof` note: the default import of an `export =` module is the package + * itself, so a bare `new` and `.loadAsync` both work on the result. */ +export function jsZip(): Promise<JSZipStatic> { + if (!load) { + load = (async () => { + try { + if (install) await install; // whatever it resolved to -- the import below surfaces a real failure with its own message + return (await import("jszip")).default; + } catch (err) { + load = null; // not cached: a later call retries once the background install has had its chance + throw err; + } + })(); + } + return load; +} diff --git a/extensions/office/src/jwt.ts b/extensions/office/src/jwt.ts new file mode 100644 index 000000000..d875634b8 --- /dev/null +++ b/extensions/office/src/jwt.ts @@ -0,0 +1,96 @@ +/** + * Minimal HS256 JWT for pithagoras-office -- config signing (minted in /editor-config) and callback + * verification share ONE secret: a random hex value the server keeps 0600 at dataDir()/jwt-secret and + * passes to the DS container as JWT_SECRET. This module is pure: no file access, no fs, no portal; + * the caller owns the secret string (per the wave-2 amendments). node:crypto only -- there is no jwt + * dependency in the monorepo lockfile. + */ +import { createHmac, timingSafeEqual } from "node:crypto"; + +function b64url(bytes: Buffer | string): string { + return Buffer.from(typeof bytes === "string" ? bytes : bytes).toString("base64url"); +} + +function b64urlDecode(text: string): Buffer | null { + if (!/^[A-Za-z0-9_-]*$/.test(text)) return null; // base64url alphabet only (no padding in jwt) + try { + return Buffer.from(text, "base64url"); + } catch { + return null; + } +} + +function sign(signingInput: string, secret: string): string { + return b64url(createHmac("sha256", secret).update(signingInput).digest()); +} + +/** HS256-sign a payload. When `ttlMs` is given, iat (now, seconds precision) and exp (now + ttl, rounded up to the + * second so the token outlives its ttl) are added automatically; when `ttlMs` is undefined NO exp claim is written + * (the token then never expires on its own). A caller-supplied iat is kept. When `purpose` is given it is embedded + * as a signed "typ" CLAIM (purpose separation, S3): every verifier of that family passes the same purpose to + * verifyJwt and refuses tokens minted for another one -- without changing the key or breaking DS-side verification, + * which ignores unknown claims. The contract shape is signJwt(payload, ttlMs?, secret?); TS forbids a required + * parameter after an optional one, so the call forms are exposed as overloads and resolved in the implementation. + * Callers type-check exactly against the contract. */ +export function signJwt(payload: Record<string, unknown>, secret: string): string; +export function signJwt(payload: Record<string, unknown>, ttlMs?: number, secret?: string, purpose?: string): string; +export function signJwt(payload: Record<string, unknown>, maybeTtlOrSecret: number | string | undefined = undefined, secret: string | undefined = undefined, purpose: string | undefined = undefined): string { + const ttlMs = typeof maybeTtlOrSecret === "number" ? maybeTtlOrSecret : undefined; + const signingSecret = typeof maybeTtlOrSecret === "string" ? maybeTtlOrSecret : (secret ?? ""); + if (signingSecret === "") throw new Error("signJwt: the HS256 secret is required"); + const header = b64url(JSON.stringify({ alg: "HS256", typ: "JWT" })); + const claims: Record<string, unknown> = { ...payload }; + if (ttlMs !== undefined) { + const nowSec = Math.floor(Date.now() / 1000); + if (claims.iat === undefined) claims.iat = nowSec; + claims.exp = nowSec + Math.ceil(ttlMs / 1000); // a negative ttl yields an already-expired token (used by tests) + } + if (purpose !== undefined && purpose !== "") claims.typ = purpose; // signed into the payload: part of the HMAC input + const body = b64url(JSON.stringify(claims)); + return `${header}.${body}.${sign(`${header}.${body}`, signingSecret)}`; +} + +/** Verify structure, pin header.alg to exactly "HS256" (anything else -- RS256, none, ... -- is null), recompute the + * HMAC and compare in constant time, then reject an expired token (exp < now, seconds). When `purpose` is given, + * the token's signed "typ" claim must equal it exactly (purpose separation) -- a validly-signed editor-config or + * fetch token can never pass as a callback of another family. Tokens without a typ claim (minted before purpose + * was introduced, or by DS itself) are accepted when NO purpose is requested, so the existing 15-minute + * verification flow and already-running containers keep working unchanged. Returns the payload object or null for + * any malformation; never throws. */ +export function verifyJwt(token: string, secret: string, purpose?: string): Record<string, unknown> | null { + const parts = token.split("."); + if (parts.length !== 3) return null; + const [headB64, bodyB64, sigB64] = parts; + const headerBuf = b64urlDecode(headB64); + const bodyBuf = b64urlDecode(bodyB64); + const sigBuf = b64urlDecode(sigB64); + if (!headerBuf || !bodyBuf || !sigBuf) return null; + let header: unknown; + try { + header = JSON.parse(headerBuf.toString("utf8")); + } catch { + return null; + } + if (typeof header !== "object" || header === null || (header as Record<string, unknown>).alg !== "HS256") return null; + const expected = sign(`${headB64}.${bodyB64}`, secret); // recompute over the EXACT transmitted segments + const expectedBuf = Buffer.from(expected, "base64url"); + if (sigBuf.length !== expectedBuf.length || !timingSafeEqual(sigBuf, expectedBuf)) return null; + let claims: unknown; + try { + claims = JSON.parse(bodyBuf.toString("utf8")); + } catch { + return null; + } + if (typeof claims !== "object" || claims === null || Array.isArray(claims)) return null; + const obj = claims as Record<string, unknown>; + if (obj.exp !== undefined) { + const exp = typeof obj.exp === "number" ? obj.exp : Number(obj.exp); + if (!Number.isFinite(exp)) return null; + if (exp < Math.floor(Date.now() / 1000)) return null; // expired + } + if (purpose !== undefined) { + const typ = obj.typ; // the SIGNATURE already covers this claim: a forger cannot strip or rewrite it + if (typeof typ !== "string" || typ !== purpose) return null; // wrong family or no claim at all + } + return obj; +} diff --git a/extensions/office/src/live.ts b/extensions/office/src/live.ts new file mode 100644 index 000000000..1cf87b0c5 --- /dev/null +++ b/extensions/office/src/live.ts @@ -0,0 +1,160 @@ +/** + * In-memory co-editing registry, shared by src/callback.ts (DS status-1 presence + the lock commands of the + * plan 5.3 state machine), src/server.ts (the /state poller and editor-config minting) and later src/ping.ts. + * + * Deliberately dependency-free -- no fs, no node:fs, no portal(), no store import: pure process memory keyed by + * docId, so every module (and every unit test) can import it anywhere without a started server half or a data + * dir. The ONE piece of external knowledge pollState needs (the document's current mtimeMs revision) is injected + * through setMetaResolver: server.ts registers `docId -> store.getDoc(docId)?.mtimeMs ?? null` once at boot, and + * until then (or in tests without a resolver) pendingRev stays null. The contract's two-argument pollState shape + * is kept exactly for that reason -- the resolver lives at module scope instead of becoming a third parameter. + * + * State is process-local by design: after a portal restart nobody is known to be editing until the DS session's + * next status-1 callback or the next editor-config mint calls setUser again, which is the correct reading of + * "closed with no changes" for a fresh process. + * + * Lock model (plan 5.3 table): an EXPLICIT flag carries the command-based entries -- status 2 "unlock after + * persist", status 4 "unlock", status 6 "keep locked" -- while status 1's row says "set users[] from body + * (lock = users.length>0)", so presence itself also counts as a lock: someone who opened the editor (a + * server-minted setUser before any callback, or a DS connect we are about to hear about) keeps the doc locked + * until they leave. pollState therefore reports `locked` when EITHER holds; unlocking never hides a user that is + * still tracked as present. Conservative on purpose: a false "still editing" badge beats a false "free to edit". + * TERMINAL CLOSES are authoritative though: status 2 (ready for saving) and status 4 (closed with no changes) + * mean nobody is connected anymore, so callback.ts clears presence AND the flag there (clearDoc) instead of + * only flipping the flag -- otherwise a server-minted user without a DS leave event would badge forever. + * + * On top of that, tracked PRESENCE carries a TTL (DEFAULT_PRESENCE_TTL_MS = 15 min, setPresenceTtl to change): + * presence is minted at editor-config time for a single shared seat and only cleared on terminal closes, so + * an abandoned open (api.js never loads, tab closed before DS connects) would otherwise hold the doc locked for + * DS and badge "still editing" -- forever. Stale entries are dropped lazily on read (usersOf/pollState); a live + * session is immune because every status-1 heartbeat re-calls setUser and pushes expiresAt forward. This adds + * expiry only -- it does NOT change lock/badge semantics: a non-expired open session still locks the doc for + * DS (the status rules above) and is reported to polling clients by pollState. Agent edits are gated purely on + * the revision, never on who is present -- an open editor neither allows nor blocks office_edit (co-editor model). + */ + +export type LiveUser = { id: string; name?: string; at: number }; + +/** A tracked entry plus its expiry (internal: usersOf returns LiveUser copies without it). + * The TTL closes the stuck-lock DoS of S2: presence is minted at editor-config time (a single shared seat, + * EDITOR_USER id "portal-user") and only cleared unconditionally on a terminal DS close -- so a failed or + * abandoned open (api.js never loads, tab closed before DS connects) left the doc locked FOREVER. With a TTL + * such a stuck entry auto-expires; a LIVE session keeps itself fresh because every status-1 heartbeat re-calls + * setUser and pushes expiresAt forward. + * 15 minutes matches CONFIG_TTL_MS (server.ts): an editor config outlives a single page-load, and the badge is + * already best-effort (the /state poller refreshes it every 5 s from this same map). */ +const DEFAULT_PRESENCE_TTL_MS = 15 * 60_000; +type PresenceRecord = LiveUser & { expiresAt: number }; + +let presenceTtlMs = DEFAULT_PRESENCE_TTL_MS; + +/** Change the presence TTL (default DEFAULT_PRESENCE_TTL_MS). Affects joins from now on; already-tracked users + * keep their existing expiry until their next heartbeat re-sets them. Tests use it to exercise expiry fast. + * Non-finite or non-positive values are ignored rather than stored (0 would expire everything immediately). */ +export function setPresenceTtl(ms: number): void { + if (Number.isFinite(ms) && ms > 0) presenceTtlMs = ms; +} + +/** The configured presence TTL in milliseconds. */ +export function getPresenceTtl(): number { + return presenceTtlMs; +} + +function isStale(u: PresenceRecord, now: number): boolean { + return u.expiresAt <= now; // inclusive: exactly-at-expiry counts as gone (clocks only ever move forward here) +} + +/** docId -> userId -> PresenceRecord. Maps (not arrays) so a re-join of the same user never duplicates. */ +const presence = new Map<string, Map<string, PresenceRecord>>(); +/** docIds explicitly locked by a callback command (statuses 1/2/4/6). */ +const locked = new Set<string>(); +/** Injected source of "the document's current mtimeMs revision"; null until server.ts registers one. */ +let resolveMeta: ((docId: string) => number | null) | null = null; + +function userMap(docId: string): Map<string, PresenceRecord> { + let users = presence.get(docId); + if (!users) { + users = new Map(); + presence.set(docId, users); + } + return users; +} + +/** Everyone currently tracked as present on the doc (join time ascending), or [] for none. Returns copies -- + * callers may hold onto them freely. Expired entries (TTL passed without a heartbeat) are dropped on sight: + * they no longer count as "editing now", so tools.ts' "editing now" annotation expires with them. */ +export function usersOf(docId: string): LiveUser[] { + const users = presence.get(docId); + if (!users) return []; + const now = Date.now(); + for (const [id, u] of [...users]) if (isStale(u, now)) users.delete(id); + return [...users.values()].map(({ id, name, at }) => ({ id, name, at })); +} + +/** Track a join (joined=true -- inserts or updates the entry, KEEPING the original `at` of an already-tracked + * user so the badge shows when they first opened it) or a leave (joined=false -- removes the id; a no-op if + * absent). Used by status-1 action handling AND by server.ts when it mints an editor config. An empty user id + * is ignored rather than stored under "". */ +export function setUser(docId: string, user: { id: string; name?: string }, joined: boolean): void { + if (!user || typeof user.id !== "string" || user.id === "") return; + const users = userMap(docId); + if (joined) { + const existing = users.get(user.id); + // `at` keeps the FIRST join (the badge shows when they opened it); expiresAt moves forward on EVERY + // join/heartbeat, so a live session never trips its own TTL while a dead one does. + users.set(user.id, { id: user.id, name: user.name ?? existing?.name, at: existing?.at ?? Date.now(), expiresAt: Date.now() + presenceTtlMs }); + } else { + users.delete(user.id); + } +} + +/** The explicit lock flag (plan 5.3: statuses 1/2/4/6 set it). */ +export function setLocked(docId: string, isLocked: boolean): void { + if (isLocked) locked.add(docId); + else locked.delete(docId); +} + +/** Whether the explicit lock flag is set for this doc. */ +export function isLocked(docId: string): boolean { + return locked.has(docId); +} + +/** + * The poll answer of the /state route (plan 5.7 tier-1 fallback, polled every 5 s by the editor page): + * `locked` -- an explicit lock flag OR at least one tracked present user (see module header for the model); + * `pendingRev` -- the document's current mtimeMs when a resolver is registered and reports it as strictly + * NEWER than clientMtimeMs (an agent revision landed since the client opened it; same number = same revision + * = not pending), else null. No resolver registered -> null. + */ +export function pollState(docId: string, clientMtimeMs: number): { locked: boolean; pendingRev: number | null } { + // usersOf filters + physically drops expired entries, so "present" here can never count a stale lock + const present = usersOf(docId).length > 0; + let pendingRev: number | null = null; + if (resolveMeta) { + const mtimeMs = resolveMeta(docId); + if (mtimeMs !== null && mtimeMs > clientMtimeMs) pendingRev = mtimeMs; + } + return { locked: present || locked.has(docId), pendingRev }; +} + +/** Register the function that answers "what is this doc's current revision mtime?" (server.ts at boot, with a + * store.getDoc lookup); pass null to clear it. Must be a pure synchronous read of the library index. */ +export function setMetaResolver(resolver: ((docId: string) => number | null) | null): void { + resolveMeta = resolver; +} + +/** Drop ALL state for one doc -- server.ts calls this after store.deleteDoc so a deleted doc never shows a + * phantom lock badge. */ +export function clearDoc(docId: string): void { + presence.delete(docId); + locked.delete(docId); +} + +/** Wipe every doc's state (and the resolver) and restore the DEFAULT presence TTL. Exported for unit tests that + * share one process; safe to call at any other time -- it only loses in-memory lock badges, nothing on disk. */ +export function resetLive(): void { + presence.clear(); + locked.clear(); + resolveMeta = null; + presenceTtlMs = DEFAULT_PRESENCE_TTL_MS; +} diff --git a/extensions/office/src/office-ui.ts b/extensions/office/src/office-ui.ts new file mode 100644 index 000000000..233830091 --- /dev/null +++ b/extensions/office/src/office-ui.ts @@ -0,0 +1,29 @@ +/** + * The browser half's pure helpers: no React, no portal, no DOM of their own -- so they can be unit-tested from + * the node side exactly as the server half's seams are. Everything that needs the portal sits in portal.tsx and web/. + */ + +/** + * The document open on the Office page, out of its address: `?doc=<id>` says it (the same convention the docs + * extension uses for `?folder=&doc=`). Encoded ids come back decoded; anything else means the list view. + */ +export function parseDocQuery(search: string): string | null { + // Tolerated as it comes from the page's address -- a query, or even an address with its path still on it. + const query = search.includes("?") ? search.slice(search.indexOf("?") + 1) : search; + const doc = new URLSearchParams(query).get("doc"); + return doc && doc !== "" ? doc : null; +} + +/** + * What POST /settings refuses on its side (400 invalid_image_tag), refused here in the input before it is sent: + * a floating image tag moves under you and breaks the co-editing keys, so the pinning warning is enforced, not + * only said. The same words of the server's rule, mirrored client-side. + */ +export function floatingTag(tag: string): boolean { + return /:(latest|stable)$/i.test(tag) || tag.endsWith("@latest"); +} + +/** How many are in a document right now, out of the polled locks map: every doc has an entry, empty when nobody. */ +export function editorsIn(locks: Record<string, { id: string; name?: string }[]> | undefined, docId: string): number { + return (locks?.[docId] ?? []).length; +} diff --git a/extensions/office/src/ooxml.ts b/extensions/office/src/ooxml.ts new file mode 100644 index 000000000..437425c40 --- /dev/null +++ b/extensions/office/src/ooxml.ts @@ -0,0 +1,558 @@ +/** + * OOXML primitives for pithagoras-office: parse docx/xlsx into text and comments, apply quote-based + * edits to a document.xml string, and rewrite the zip around changed parts. All functions here are + * pure over Buffers/strings (no fs, no settings, no network), per the contracts file; store.ts owns + * disk, tools.ts owns the tool surface. + * + * ZIP I/O is a small synchronous reader/writer on node:zlib rather than jszip: the contract gives + * parseDocx/parseXlsx/rewriteZip synchronous signatures (the ping diff and the office_* tools run + * them inline), and jszip only exposes async load/generate. The writer keeps entry order and is + * content-preserving -- every untouched entry re-extracts byte-identical -- but recompresses, so an + * output zip is NOT compression-byte-identical to its input (accepted deviation from the plan's + * "byte-for-byte", noted in .office-contracts.md). node builtins only; jszip stays for templates. + */ +import { deflateRawSync, inflateRawSync } from "node:zlib"; + +export type Comment = { id: string; author: string; date?: string; text: string }; + +/** One agent edit: quote a unique passage of the document text and say what replaces it. */ +export type EditOp = { quote: string; replacement: string }; + +function ooxmlError(message: string, code: string): Error { + return Object.assign(new Error(message), { code }); +} + +/* ------------------------------------------------------------------ */ +/* minimal sync zip reader/writer (stored + deflate) */ +/* ------------------------------------------------------------------ */ + +type ZipEntry = { + name: string; + method: number; // 0 stored, 8 deflated + crc: number; + csize: number; + usize: number; + lfhOffset: number; + modTime: number; + modDate: number; +}; + +const EOCD_SIG = 0x06054b50; +const CD_SIG = 0x02014b50; +const LFH_SIG = 0x04034b50; +const ZIP64 = 0xffffffff; + +/** Entry directory of a zip: central-directory records in file order. Throws {code:"invalid_zip"}. */ +function readZip(buf: Buffer): ZipEntry[] { + if (buf.length < 22) throw ooxmlError("not a zip archive (too small)", "invalid_zip"); + // EOCD lives in the last 22+65535 bytes. + const from = Math.max(0, buf.length - 22 - 65535); + let eocd = -1; + for (let i = buf.length - 22; i >= from; i--) if (buf.readUInt32LE(i) === EOCD_SIG) { eocd = i; break; } + if (eocd < 0) throw ooxmlError("not a zip archive (no end of central directory)", "invalid_zip"); + const count = buf.readUInt16LE(eocd + 10); + const cdSize = buf.readUInt32LE(eocd + 12); + const cdOffset = buf.readUInt32LE(eocd + 16); + if (count === 0xffff || cdSize === ZIP64 || cdOffset === ZIP64) throw ooxmlError("zip64 archives are not supported", "invalid_zip"); + const entries: ZipEntry[] = []; + let totalUncompressed = 0; // running sum of the ADVERTISED expansion of every entry (archive cap below) + let p = cdOffset; + for (let n = 0; n < count; n++) { + if (p + 46 > buf.length || buf.readUInt32LE(p) !== CD_SIG) throw ooxmlError("corrupt zip central directory", "invalid_zip"); + const method = buf.readUInt16LE(p + 10); + const csize = buf.readUInt32LE(p + 20); + const usize = buf.readUInt32LE(p + 24); + if (method === ZIP64 || csize === ZIP64 || usize === ZIP64) throw ooxmlError("zip64 archives are not supported", "invalid_zip"); + totalUncompressed += usize; // BEFORE the entry is retained, so a hostile central directory cannot accumulate unbounded expansion + if (totalUncompressed > MAX_TOTAL_BYTES) throw ooxmlError(`combined uncompressed size exceeds the ${MAX_TOTAL_BYTES / (1024 * 1024)} MiB archive cap`, "invalid_zip"); + const nameLen = buf.readUInt16LE(p + 28); + const extraLen = buf.readUInt16LE(p + 30); + const commentLen = buf.readUInt16LE(p + 32); + const lfhOffset = buf.readUInt32LE(p + 42); + entries.push({ + name: buf.toString("utf8", p + 46, p + 46 + nameLen), + method, + crc: buf.readUInt32LE(p + 16), + csize, + usize, + lfhOffset, + modTime: buf.readUInt16LE(p + 12), + modDate: buf.readUInt16LE(p + 14), + }); + p = p + 46 + nameLen + extraLen + commentLen; + } + return entries; +} + +// Hard cap on the uncompressed size of any single zip entry: a corrupt or hostile central directory could +// otherwise advertise a multi-GB expansion (stored or deflated) that the portal would commit into memory. +// 256 MiB is far above anything real in docx/xlsx parts we ever read back. +const MAX_ENTRY_BYTES = 256 * 1024 * 1024; + +// Same guard at the archive level: readZipParts retains EVERY central-directory entry, so a hostile archive could +// advertise several just-under-cap entries -- or thousands of small ones -- whose combined uncompressed size is far +// beyond anything a docx/xlsx reader would ever touch. 1 GiB total is still far beyond real files; reject the whole +// archive when the advertised sum crosses it (checked in readZip, before any entry is retained). +const MAX_TOTAL_BYTES = 1024 * 1024 * 1024; + +/** Uncompressed payload of one entry (payload length comes from the central directory, so local data descriptors are fine). */ +function readEntryData(buf: Buffer, e: ZipEntry): Buffer { + if (e.lfhOffset + 30 > buf.length || buf.readUInt32LE(e.lfhOffset) !== LFH_SIG) throw ooxmlError(`corrupt zip local header for ${e.name}`, "invalid_zip"); + const nameLen = buf.readUInt16LE(e.lfhOffset + 26); + const extraLen = buf.readUInt16LE(e.lfhOffset + 28); + const start = e.lfhOffset + 30 + nameLen + extraLen; + if (start + e.csize > buf.length) throw ooxmlError(`truncated zip payload for ${e.name}`, "invalid_zip"); + const raw = buf.subarray(start, start + e.csize); + if (e.usize > MAX_ENTRY_BYTES) throw ooxmlError(`zip entry ${e.name} exceeds the ${MAX_ENTRY_BYTES / (1024 * 1024)} MiB per-entry cap`, "invalid_zip"); + let out: Buffer; + if (e.method === 0) out = raw; + else if (e.method === 8) { try { out = inflateRawSync(raw, { maxOutputLength: Math.max(e.usize, 1) }); } catch { throw ooxmlError(`bad deflate data for ${e.name}`, "invalid_zip"); } } + else throw ooxmlError(`unsupported zip compression method ${e.method} for ${e.name}`, "invalid_zip"); + if (out.length !== e.usize) throw ooxmlError(`size mismatch for ${e.name}`, "invalid_zip"); + // CRC-32 of the DECODED payload vs the central directory's record. Without this, bit-rot / partial-write + // corruption (the most common real-world kind -- a flipped stored-entry byte or a deflate stream that still + // inflates to the advertised length) would parse silently and rewriteZip would re-persist the corrupt bytes + // with their original wrong CRC, masking the damage inside the library. Cheap: `out` is already materialized. + if (crc32(out) !== e.crc) throw ooxmlError(`CRC mismatch for ${e.name}: the stored entry data does not match its checksum`, "invalid_zip"); + return out; +} + +const CRC_TABLE = (() => { + const table = new Uint32Array(256); + for (let n = 0; n < 256; n++) { let c = n; for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1; table[n] = c; } + return table; +})(); + +function crc32(data: Buffer): number { + let c = 0xffffffff; + for (let i = 0; i < data.length; i++) c = CRC_TABLE[(c ^ data[i]) & 0xff] ^ (c >>> 8); + return (c ^ 0xffffffff) >>> 0; +} + +function dosDateTime(d: Date): { time: number; date: number } { + return { + time: (d.getHours() << 11) | (d.getMinutes() << 5) | (d.getSeconds() >> 1), + date: ((Math.max(1980, d.getFullYear()) - 1980) << 9) | ((d.getMonth() + 1) << 5) | d.getDate(), + }; +} + +/** Fresh zip from named entries (order kept). Deflate for non-empty data, stored otherwise. + * Exported so scripts/make-fixtures.mjs can build the committed fixtures THROUGH this writer -- the + * fixtures then double as round-trip material for our own reader/writer pair. */ +export function writeZip(entries: { name: string; data: Buffer; time?: { time: number; date: number } }[]): Buffer { + const now = dosDateTime(new Date()); + const chunks: Buffer[] = []; + const cd: Buffer[] = []; + let offset = 0; + for (const entry of entries) { + const time = entry.time ?? now; + const name = Buffer.from(entry.name, "utf8"); + const data = entry.data; + const deflate = data.length > 0; + const payload = deflate ? deflateRawSync(data, { level: 6 }) : Buffer.alloc(0); + const crc = crc32(data); + // Local file header. UTF-8 names get the flag bit set; no extra field, no data descriptor. + const lfh = Buffer.alloc(30); + lfh.writeUInt32LE(LFH_SIG, 0); + lfh.writeUInt16LE(20, 4); // version needed + lfh.writeUInt16LE(/[^\x00-\x7f]/.test(entry.name) ? 0x800 : 0, 6); + lfh.writeUInt16LE(deflate ? 8 : 0, 8); + lfh.writeUInt16LE(time.time, 10); + lfh.writeUInt16LE(time.date, 12); + lfh.writeUInt32LE(crc, 14); + lfh.writeUInt32LE(payload.length, 18); + lfh.writeUInt32LE(data.length, 22); + lfh.writeUInt16LE(name.length, 26); + lfh.writeUInt16LE(0, 28); + chunks.push(lfh, name, payload); + const rec = Buffer.alloc(46); + rec.writeUInt32LE(CD_SIG, 0); + rec.writeUInt16LE(20, 4); // version made by + rec.writeUInt16LE(20, 6); // version needed + rec.writeUInt16LE(lfh.readUInt16LE(6), 8); + rec.writeUInt16LE(deflate ? 8 : 0, 10); + rec.writeUInt16LE(time.time, 12); + rec.writeUInt16LE(time.date, 14); + rec.writeUInt32LE(crc, 16); + rec.writeUInt32LE(payload.length, 20); + rec.writeUInt32LE(data.length, 24); + rec.writeUInt16LE(name.length, 28); + rec.writeUInt16LE(0, 30); // extra + rec.writeUInt16LE(0, 32); // comment + rec.writeUInt16LE(0, 34); // disk number + rec.writeUInt16LE(0, 36); // internal attrs + rec.writeUInt32LE(0, 38); // external attrs + rec.writeUInt32LE(offset, 42); + cd.push(rec, name); + offset += lfh.length + name.length + payload.length; + } + const cdBuf = Buffer.concat(cd); + const eocd = Buffer.alloc(22); + eocd.writeUInt32LE(EOCD_SIG, 0); + eocd.writeUInt16LE(entries.length, 8); + eocd.writeUInt16LE(entries.length, 10); + eocd.writeUInt32LE(cdBuf.length, 12); + eocd.writeUInt32LE(offset, 16); + return Buffer.concat([...chunks, cdBuf, eocd]); +} + +/** All file parts of a zip as uncompressed bytes; duplicate names: last one wins (jszip behavior). */ +function readZipParts(buf: Buffer): Record<string, Buffer> { + const parts: Record<string, Buffer> = {}; + for (const entry of readZip(buf)) parts[entry.name] = readEntryData(buf, entry); + return parts; +} + +/* ------------------------------------------------------------------ */ +/* small targeted XML helpers (no full DOM -- only the tags these */ +/* functions own) */ +/* ------------------------------------------------------------------ */ + +function decodeEntities(s: string): string { + return s.replace(/&#x([0-9a-fA-F]+);|&#([0-9]+);|&([a-zA-Z][a-zA-Z0-9]*);/g, (match, hex, dec, name) => { + if (hex !== undefined) return String.fromCodePoint(parseInt(hex, 16)); + if (dec !== undefined) return String.fromCodePoint(parseInt(dec, 10)); + switch (name) { case "amp": return "&"; case "lt": return "<"; case "gt": return ">"; case "quot": return '"'; case "apos": return "'"; } + return match; // unknown named entity: keep as-is rather than drop text + }); +} + +/** Escape a string for use as w:t / t element content. */ +function escapeXmlText(s: string): string { + return s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">"); +} + +/** Attribute name="value" pairs of a tag (double-quoted, OOXML style); keys keep their local part (w:id -> id). */ +function parseAttributes(tag: string): Record<string, string> { + const attrs: Record<string, string> = {}; + for (const m of tag.matchAll(/([A-Za-z_][\w.:-]*)="([^"]*)"/g)) attrs[m[1].slice(m[1].lastIndexOf(":") + 1)] = decodeEntities(m[2]); + return attrs; +} + +type Span = { tagStart: number; tagEnd: number; text: string }; // absolute offsets in the XML string + +/** Every w:t of one paragraph-content string as decoded-text spans (ordered, so their concatenation is the paragraph text). */ +function wtSpans(content: string, base: number): Span[] { + const spans: Span[] = []; + for (const m of content.matchAll(/<w:t(?:\s[^>]*)?>([\s\S]*?)<\/w:t>/g)) { + if (m.index === undefined) continue; + spans.push({ tagStart: base + m.index, tagEnd: base + m.index + m[0].length, text: decodeEntities(m[1]) }); + } + return spans; +} + +type Para = { contentStart: number; contentEnd: number }; // paragraph-content offsets in the XML string (self-closing w:p: equal) + +/** Every top-level w:p of a document.xml body in order. Nested paragraphs (text boxes, w:txbxContent) are out of scope + * by contract ("w:p > w:r > w:t"); such content would be read as the outer paragraph's trailing run and editing it is unsupported. */ +function findParagraphs(xml: string): Para[] { + const paras: Para[] = []; + for (const m of xml.matchAll(/<w:p(?:\s[^>]*?)?(?:\/>|>([\s\S]*?)<\/w:p>)/g)) { + if (m.index === undefined) continue; + const content = m[1]; + paras.push(content === undefined ? { contentStart: 0, contentEnd: 0 } : { contentStart: m.index + m[0].indexOf(">") + 1, contentEnd: m.index + m[0].length - "</w:p>".length }); + } + return paras; +} + +/** Decoded text of one w:p's content (empty string when the paragraph has no runs -- empty paragraphs are kept on purpose). */ +function paragraphText(content: string): string { + let out = ""; + for (const m of content.matchAll(/<w:t(?:\s[^>]*)?>([\s\S]*?)<\/w:t>/g)) out += decodeEntities(m[1]); + return out; +} + +/* ------------------------------------------------------------------ */ +/* docx */ +/* ------------------------------------------------------------------ */ + +/** + * Extract a docx: paragraph texts (one entry per w:p, in order, empty paragraphs kept as "") joined with + * "\n" into `text`, plus the word/comments.xml comments. Missing word/document.xml throws {code:"missing_part"}; + * a missing comments part is simply no comments. Runs are read as plain text: w:tab/w:br and other run children + * contribute nothing (simplification per contract). + */ +export function parseDocx(buf: Buffer): { text: string; paragraphs: string[]; comments: Comment[] } { + const parts = readZipParts(buf); + const document = parts["word/document.xml"]; + if (!document) throw ooxmlError("docx has no word/document.xml", "missing_part"); + const xml = document.toString("utf8"); + const paragraphs: string[] = []; + for (const para of findParagraphs(xml)) { + const content = para.contentStart === 0 && para.contentEnd === 0 ? "" : xml.slice(para.contentStart, para.contentEnd); + paragraphs.push(paragraphText(content)); + } + let comments: Comment[] = []; + if (parts["word/comments.xml"]) comments = parseComments(parts["word/comments.xml"].toString("utf8")); + return { text: paragraphs.join("\n"), paragraphs, comments }; +} + +/** The w:comment elements of a comments.xml; each comment's text is its w:p/w:r/w:t runs joined in order. */ +function parseComments(xml: string): Comment[] { + const comments: Comment[] = []; + for (const m of xml.matchAll(/<w:comment\b([^>]*)>([\s\S]*?)<\/w:comment>/g)) { + if (m.index === undefined) continue; + const attrs = parseAttributes(m[1]); + let text = ""; + for (const t of m[2].matchAll(/<w:t(?:\s[^>]*)?>([\s\S]*?)<\/w:t>/g)) text += decodeEntities(t[1]); + const comment: Comment = { id: attrs["id"] ?? "", author: attrs["author"] ?? "", text }; + if (attrs["date"] !== undefined) comment.date = attrs["date"]; + comments.push(comment); + } + return comments; +} + +/** + * Apply quote-based edits to a document.xml string, in order -- op n+1 quotes the document as op n left it. + * A quote must occur exactly once (distinct occurrences) in the concatenated paragraph text -- the same text + * parseDocx would extract (exact substring, case-sensitive). The replacement lands where the quote sat: the + * first affected w:t keeps the run's leftover prefix plus the new text (plain runs), fully covered w:t elements + * are dropped, and a partially covered final w:t keeps its suffix. Surrounding XML is untouched; rewritten + * w:t tags get xml:space="preserve" when their content has leading or trailing spaces so Word does not strip them. + * Throws {code:"quote_not_found"} (also for an empty quote) and {code:"quote_ambiguous"}. + */ +export function editDocx(documentXml: string, ops: EditOp[]): string { + let xml = documentXml; + for (const op of ops) xml = applyEdit(xml, op); + return xml; +} + +function applyEdit(xml: string, op: EditOp): string { + const paras = findParagraphs(xml); + const spanLists = paras.map((p) => wtSpans(p.contentStart === 0 && p.contentEnd === 0 ? "" : xml.slice(p.contentStart, p.contentEnd), p.contentStart)); + // The virtual text: each paragraph's runs in order, paragraphs joined with "\n" (matching parseDocx). Map every + // w:t span back to its offsets in that text. + const paraRanges: { start: number; length: number }[] = []; + let cursor = 0; + paras.forEach((_, i) => { + const length = spanLists[i].reduce((n, s) => n + s.text.length, 0); + paraRanges.push({ start: cursor, length }); + cursor += length + (i < paras.length - 1 ? 1 : 0); // the joining "\n" has no XML behind it + }); + const virtual = paras.map((_, i) => spanLists[i].map((s) => s.text).join("")).join("\n"); + + if (op.quote === "") throw ooxmlError("quote must not be empty", "quote_not_found"); + const at = virtual.indexOf(op.quote); + if (at === -1) throw ooxmlError("quote_not_found: the quoted passage is not in the document", "quote_not_found"); + if (virtual.indexOf(op.quote, at + 1) !== -1) throw ooxmlError("quote_ambiguous: the quoted passage occurs more than once", "quote_ambiguous"); + + const qEnd = at + op.quote.length; + // Tag-level edits in the xml string: for each w:t span the quote touches, rewrite its content. + const edits: { start: number; end: number; text: string }[] = []; + let firstDone = false; + paras.forEach((para, i) => { + if (para.contentStart === para.contentEnd) return; + const pLo = Math.max(at, paraRanges[i].start); + const pHi = Math.min(qEnd, paraRanges[i].start + paraRanges[i].length); + if (pLo >= pHi) return; // this paragraph is outside the quote + let spanOffset = paraRanges[i].start; // where this span's text starts in the virtual string + for (const span of spanLists[i]) { + const spanStart = spanOffset; + const spanEnd = spanOffset + span.text.length; + const lo = Math.max(pLo, spanStart); + const hi = Math.min(pHi, spanEnd); + spanOffset = spanEnd; + if (lo >= hi) continue; // span untouched by this quote + const prefix = span.text.slice(0, lo - spanStart); // of the first touched span, what stays before the quote + const suffix = span.text.slice(hi - spanStart); // of a last touched span, what stays after the quote (empty when it ends exactly at the span's end) + let writeContent: string; + if (!firstDone) { writeContent = prefix + op.replacement + suffix; firstDone = true; } // the replacement lands in the first touched span, its leftover text around it stays + else writeContent = suffix; // earlier spans already consumed everything up to `hi`; a later touched span keeps only its tail + if (writeContent === "") edits.push({ start: span.tagStart, end: span.tagEnd, text: "" }); // fully covered: drop the w:t element + else { + const preserve = /^[ \t\r\n]|[ \t\r\n]$/.test(writeContent); + edits.push({ start: span.tagStart, end: span.tagEnd, text: `<w:t${preserve ? ' xml:space="preserve"' : ""}>${escapeXmlText(writeContent)}</w:t>` }); + } + } + }); + // A quote that touches no w:t span at all (e.g. one spanning only paragraph breaks) passed the uniqueness + // check above but would leave the XML unchanged -- persisting it as a successful edit is wrong, so reject it. + if (!firstDone) throw ooxmlError("quote_not_found: the quoted passage covers no text runs (paragraph breaks only)", "quote_not_found"); + for (const edit of edits.sort((a, b) => b.start - a.start)) xml = xml.slice(0, edit.start) + edit.text + xml.slice(edit.end); + return xml; +} + +/* ------------------------------------------------------------------ */ +/* xlsx */ +/* ------------------------------------------------------------------ */ + +/** Resolve a workbook rels Target against xl/: absolute ("/xl/...") or relative ("worksheets/sheet1.xml"). */ +function resolveTarget(target: string): string { + const pieces = target.startsWith("/") ? target.slice(1).split("/") : ["xl", ...target.split("/")]; + const parts: string[] = []; + for (const piece of pieces) if (!piece || piece === ".") continue; else if (piece === "..") parts.pop(); else parts.push(piece); + return parts.join("/"); +} + +function columnOf(ref: string): number { // "AB12" -> 27 (zero-based), -1 when the ref has no letters + let col = 0; + for (const ch of ref) { if (ch >= "A" && ch <= "Z") col = col * 26 + (ch.charCodeAt(0) - 64); else break; } + return col === 0 ? -1 : col - 1; +} + +/** + * Extract an xlsx: sheet names in workbook order, and per sheet a dense grid of cell strings -- sparse cells + * are filled to the farthest used row/column with "" (plan simplicity note: grids, not sparse maps). The + * materialized grid is bounded (see MAX_GRID_ROWS/MAX_GRID_COLS/MAX_SHEET_CELLS/MAX_TOTAL_GRID_CELLS): a sheet + * that would densify past its share of the cell budget gets only the affordable leading rows, and truncatedSheets + * names it so callers can say "those far-out cells were not read". Values: + * sharedStrings lookups (t="s"), inline strings (t="inlineStr"), and stored text as-is for numbers/formulas/ + * booleans ("keep stored text"). Cells without an r= reference are skipped. A missing workbook or referenced + * sheet part throws {code:"missing_part"}. + */ +export function parseXlsx(buf: Buffer): { sheetNames: string[]; sheets: Record<string, string[][]>; truncatedSheets: Record<string, boolean> } { + const parts = readZipParts(buf); + const workbook = parts["xl/workbook.xml"]; + if (!workbook) throw ooxmlError("xlsx has no xl/workbook.xml", "missing_part"); + const relsXml = parts["xl/_rels/workbook.xml.rels"]?.toString("utf8") ?? ""; + const targets: Record<string, string> = {}; + for (const m of relsXml.matchAll(/<Relationship\b([^>]*?)\/?>/g)) { + if (m.index === undefined) continue; + const attrs = parseAttributes(m[1]); + if (attrs["Id"] && attrs["Target"] && attrs["TargetMode"] !== "External") targets[attrs["Id"]] = attrs["Target"]; + } + let sharedStrings: string[] = []; + if (parts["xl/sharedStrings.xml"]) { + for (const m of parts["xl/sharedStrings.xml"].toString("utf8").matchAll(/<si(?:\s[^>]*?)?(?:\/>|>([\s\S]*?)<\/si>)/g)) { + const content = m[1] ?? ""; + let text = ""; + for (const t of content.matchAll(/<t(?:\s[^>]*)?>([\s\S]*?)<\/t>/g)) text += decodeEntities(t[1]); + sharedStrings.push(text); + } + } + const sheetNames: string[] = []; + const sheets: Record<string, string[][]> = {}; + const truncatedSheets: Record<string, boolean> = {}; + const gridBudget: { left: number } = { left: MAX_TOTAL_GRID_CELLS }; // decremented as each sheet densifies + for (const m of workbook.toString("utf8").matchAll(/<sheet\b([^>]*)\/?>/g)) { + if (m.index === undefined) continue; + const attrs = parseAttributes(m[1]); + const name = attrs["name"] ?? ""; + sheetNames.push(name); + sheets[name] = []; + const target = attrs["id"] !== undefined ? targets[attrs["id"]] : undefined; + if (target === undefined) continue; // unresolvable sheet: listed by name, no grid + const sheetPart = parts[resolveTarget(target)]; + if (!sheetPart) throw ooxmlError(`xlsx is missing the worksheet part ${resolveTarget(target)} for sheet "${name}"`, "missing_part"); + const flags: { truncated?: boolean } = {}; + // The grid budget is per ARCHIVE (every sheet shares it) so a workbook cannot multiply the allocation + // across many sheets; see MAX_TOTAL_GRID_CELLS below. + sheets[name] = parseSheet(sheetPart.toString("utf8"), sharedStrings, gridBudget, flags); + if (flags.truncated) truncatedSheets[name] = true; + } + return { sheetNames, sheets, truncatedSheets }; +} + +// Dense-grid caps: a hostile worksheet can reference absurdly distant cells (1M-row x 16K-col class); without +// bounds the grid allocation below is unbounded. Cap both axes; skipped far cells are reported as truncation. +const MAX_GRID_ROWS = 20000; +const MAX_GRID_COLS = 500; +// The AXIS caps alone do not bound MEMORY: a sheet with just two cells (one at row 20000, one near column 500) +// still densifies to the PRODUCT maxRow x maxCol (~10M cell slots, hundreds of MB per measured run), and there +// is no cap on sheet count. Cap the materialized product per sheet AND the combined grid across the archive -- +// the same "per-unit / total" shape as MAX_ENTRY_BYTES (256 MiB) / MAX_TOTAL_BYTES (1 GiB), in CELLS instead +// of bytes (256*1024 cells/sheet, 1024*1024 cells/archive ~= a few MB of JS arrays each). Rows beyond the +// affordable height are not allocated; the sheet is flagged in truncatedSheets so callers surface it honestly. +const MAX_SHEET_CELLS = 256 * 1024; +const MAX_TOTAL_GRID_CELLS = 1024 * 1024; +function parseSheet(xml: string, sharedStrings: string[], budget: { left: number }, flags?: { truncated?: boolean }): string[][] { + const sheetData = xml.match(/<sheetData(?:\s[^>]*)?>([\s\S]*?)<\/sheetData>/); + if (!sheetData) return []; + let maxRow = 0; + let maxCol = 0; + let truncated = false; + const cells: Record<string, string> = {}; // "row,col" -> cell text + for (const m of sheetData[1].matchAll(/<c\b([^>]*?)(?:\/>|>([\s\S]*?)<\/c>)/g)) { + if (m.index === undefined) continue; + const attrs = parseAttributes(m[1]); + if (!attrs["r"]) continue; // no reference: position unknown, skip (kept simple per plan) + const row = Number.parseInt(attrs["r"].replace(/^[A-Z]+/, ""), 10); + const col = columnOf(attrs["r"]); + if (col < 0 || !Number.isFinite(row) || row < 1) continue; + if (row > MAX_GRID_ROWS || col >= MAX_GRID_COLS) { truncated = true; continue; } // grid cap, reported by the caller + let value = ""; + const inner = m[2] ?? ""; + if (attrs["t"] === "inlineStr") { + for (const t of inner.matchAll(/<t(?:\s[^>]*)?>([\s\S]*?)<\/t>/g)) value += decodeEntities(t[1]); + } else { + const v = inner.match(/<v(?:\s[^>]*)?>([\s\S]*?)<\/v>/); + if (v) value = attrs["t"] === "s" ? (sharedStrings[Number.parseInt(v[1], 10)] ?? "") : decodeEntities(v[1]); // numbers/formulas/booleans: stored text as-is + } + cells[`${row},${col}`] = value; + if (row > maxRow) maxRow = row; + if (col >= maxCol) maxCol = col + 1; + } + // Bound the MATERIALIZED product before allocating: shrink the height so rows x maxCol stays under the + // per-sheet cell cap AND what the shared archive budget can still afford; anything beyond is truncation. + let rows = maxRow; + if (rows > 0 && maxCol > 0) { + const perSheetRows = Math.floor(MAX_SHEET_CELLS / maxCol); + const affordableRows = budget.left > 0 ? Math.floor(budget.left / maxCol) : 0; + const limit = Math.min(perSheetRows, affordableRows); + if (limit < rows) { rows = limit; truncated = true; } // the far rows exist in the file but are not read + budget.left -= rows * maxCol; + } + const grid: string[][] = []; + for (let r = 0; r < rows; r++) { + const line: string[] = []; + for (let c = 0; c < maxCol; c++) line.push(cells[`${r + 1},${c}`] ?? ""); + grid.push(line); + } + if (truncated) flags && (flags.truncated = true); + return grid; +} + +/* ------------------------------------------------------------------ */ +/* zip rewriting + comment diffing */ +/* ------------------------------------------------------------------ */ + +function asBuffer(value: string | Buffer): Buffer { + return typeof value === "string" ? Buffer.from(value, "utf8") : value; +} + +/** + * Rewrite a zip with named parts replaced or added. Part matching is CASE-INSENSITIVE (and change keys that + * collide only by case are deduped last-one-wins, jszip behavior): OOXML part names are treated case-tolerantly + * in several of Word/DS' code paths, so a key like "WORD/DOCUMENT.XML" over an archive holding word/document.xml + * OVERWRITES the existing part IN PLACE (its original name is kept) instead of creating a shadow duplicate that + * could win on load. Accepted deviation (noted in the contracts file): the output is CONTENT-preserving -- every + * untouched entry re-extracts byte-identical to its original uncompressed bytes, entry order is kept where this + * writer allows (replacements stay put, new parts are appended) -- but the zip as a whole is NOT + * compression-byte-identical to the input: unchanged deflate entries are recompressed and rewritten with fresh + * timestamps for added parts. Throws {code:"invalid_zip"} on unreadable archives. + */ +export function rewriteZip(buf: Buffer, changes: Record<string, string | Buffer>): Buffer { + const entries = readZip(buf); + // Change keys that differ from each other only in case would emit one shadow part for the same logical name -- + // keep the LAST of them (jszip "last one wins") and let it match the existing entry case-insensitively below. + const norm: Record<string, string | Buffer> = {}; + for (const [name, value] of Object.entries(changes)) { + for (const k of Object.keys(norm)) if (k.toLowerCase() === name.toLowerCase()) delete norm[k]; + norm[name] = value; + } + const changeLower = new Map<string, string>(); // lowercased part name -> the surviving change key + for (const k of Object.keys(norm)) changeLower.set(k.toLowerCase(), k); + const applied = new Set<string>(); // lowercase names consumed by a change (an input SHADOW pair gets the data once) + const out: { name: string; data: Buffer; time?: { time: number; date: number } }[] = []; + for (const entry of entries) { + let data = readEntryData(buf, entry); + const key = changeLower.get(entry.name.toLowerCase()); + if (key !== undefined && !applied.has(key.toLowerCase())) { // exact match OR differs only by case + data = asBuffer(norm[key]); + applied.add(key.toLowerCase()); + } + out.push({ name: entry.name, data, time: { time: entry.modTime, date: entry.modDate } }); + } + for (const [name, value] of Object.entries(norm)) if (!applied.has(name.toLowerCase())) out.push({ name, data: asBuffer(value) }); + return writeZip(out); +} + +/** + * Comments that `next` has which `prev` does not, by id -- an id only in `next` is added. The same id with + * changed text or author is an UPDATE and deliberately NOT reported as added (the wake diff wants new threads, + * not reworded ones). Order of `next` is kept. + */ +export function diffComments(prev: Comment[], next: Comment[]): { added: Comment[] } { + const known = new Set(prev.map((c) => c.id)); + return { added: next.filter((c) => !known.has(c.id)) }; +} diff --git a/extensions/office/src/ping.ts b/extensions/office/src/ping.ts new file mode 100644 index 000000000..b64430b12 --- /dev/null +++ b/extensions/office/src/ping.ts @@ -0,0 +1,258 @@ +/** + * Wake on comment mention (plan 5.4). Server-side only. + * + * On every docx persist (callback status 2/6 fires the `onPersisted` hooks), the saved buffer is + * re-parsed (`parseDocx`) and its word/comments.xml comments are diffed against the last-seen set + * kept in `<root>/state/<docId>.json`. Each NEW comment is scanned for mentions (grammar below); + * per resolved mention, per target chat: a grant on this one doc (deduped by comment id), an audit + * line, a queued wake text (`officeWakeMessage`), and `chats.tell(chatId, {type:"ping"})` so the + * target's pi extension drains the queue off its UI_RECEIVE bell (drain side = T-TOOLS' factory, + * plan 5.4 step 4; this module only QUEUES). The seen set is written back AFTER processing. + * + * First-save decision (plan leaves it open): an IMPORTED document arrives with its own comment history -- importFile records + * the file's comment ids in DocMeta.importedBaseline AT IMPORT TIME, and a persist with NO prior state file baselines exactly + * those ids WITHOUT resolving mentions (they predate this library and must not ring bells). A comment that appears AFTER the + * import but BEFORE the first save is NEW to this library and IS processed on that first save, so its wake is not lost. When + * the source did not parse at import the field is absent and we fall back to baselining whatever the first save holds. + * A CREATED blank document gets no baseline: its seen set starts empty, so an @mention typed during the very first editing + * session is processed on its first save like any later one. (store.ts marks imported documents with DocMeta.imported; see importFile / scanWorkspace.) + * Corrupt/undecodable state file: logged, previous set treated as EMPTY (every current comment is + * then "new", its mentions processed once; the immediately-following write restores a sane file). + * Erring toward notifying beats silently dropping a mention after a corruption event. A re-send of + * an already-seen comment adds nothing (pure id diff), so normal re-saves never double-ping or + * double-grant. + * + * v1 wakes docx ONLY: xlsx comments live in xl/commentsN.xml + VML anchors (plan 5.4 wording + * correction -- "xlsx has no comments" was false), deferred to M4, so the hook is a no-op for + * non-docx docs. onPersisted hands over (docId, buffer) only, so this module resolves the meta + * itself via store.getDoc(docId, root) and checks meta.ext. + */ +import fs from "node:fs"; +import path from "node:path"; +import { portal } from "./host"; +import { addGrant as storeAddGrant, getDoc } from "./store"; +import { parseDocx, diffComments, type Comment } from "./ooxml"; +import { onPersisted } from "./callback"; + +/** A line the audit seam records per resolved mention (subject = doc title; reason carries what happened; chatId = woken home). T-WIRE may swap in a real audit sink later. */ +export type PingAuditEntry = { subject: string; reason: string; chatId: string }; + +/** Injection seams for wirePing. Every function form is LAZY by design: the defaults read portal() only when actually used, and tests pass plain fns so they never reach the real portal. `root` overrides dataDir() for the store/state layout. */ +export type PingDeps = { + root?: string; + folders?: () => { path: string; name: string; kind: "agent" | "project" | "chat"; home?: boolean }[]; + inFolder?: (folderPath: string) => { id: string; title?: string }[]; + tell?: (chatId: string, data: unknown) => Promise<boolean> | boolean; + record?: (entry: PingAuditEntry) => void; + log?: (line: string) => void; +}; + +/** One queued wake entry, docs' `wakes` pattern: text is what pi is handed, details ride along for the drain side. */ +export type WakeEntry = { text: string; details: { docId: string; commentId: string } }; + +let cfg: PingDeps = {}; +let hookRegistered = false; +/** chatId -> wake entries queued server-side (docs' `wakes` pattern). Drained by the office pi extension on UI_RECEIVE. */ +const wakes = new Map<string, WakeEntry[]>(); + +function root(): string { + return cfg.root ?? portal().dataDir(); +} +function folders(): { path: string; name: string; kind: "agent" | "project" | "chat"; home?: boolean }[] { + return (cfg.folders ?? (() => portal().folders.list()))(); +} +function inFolder(p: string): { id: string; title?: string }[] { + return (cfg.inFolder ?? ((fp) => portal().chats.inFolder(fp)))(p); +} +function logLine(line: string): void { + (cfg.log ?? ((l) => portal().log(l)))(line); +} +function record(entry: PingAuditEntry): void { + // Lazy default is a one-line log (T-WIRE may pass a real audit sink later via wirePing({record})). + (cfg.record ?? ((e: PingAuditEntry) => logLine(`office ping: ${e.subject}: ${e.reason} (woken ${e.chatId})`)))(entry); +} + +/** server.ts calls ONCE at boot. Later calls (unit tests) only update the seams -- the onPersisted hook is registered exactly once per process. */ +export function wirePing(opts?: PingDeps): void { + if (opts) cfg = { ...cfg, ...opts }; + if (!hookRegistered) { + hookRegistered = true; + onPersisted(handlePersisted); + } +} + +/** Test seam: drop the module-level seams, hook flag and queued wakes. */ +export function __resetPingForTests(): void { + cfg = {}; + hookRegistered = false; + wakes.clear(); +} + +/** Remove and return what is queued for a chat (the office pi extension calls this on its UI_RECEIVE bell, docs' hearPings pattern). */ +export function drainWakeQueue(chatId: string): WakeEntry[] { + const entries = wakes.get(chatId); + if (!entries) return []; + wakes.delete(chatId); + return entries; +} + +/** The exact plan-5.4 wake text shape, following docs' wakeMessage(): doc title, quoted comment (every line prefixed "> "), and the grant sentence with the office_* pointer -- "given access", NOT "for this task": the added grants are permanent in v1 (per-comment expiry is deferred), so the text must not promise a scope that does not exist. The doc id is NOT embedded in the text: it rides in WakeEntry.details ({docId, commentId}) per plan 5.4 step 2, which the drain side hands to pi's sendMessage. `author` is included because an office mention is one specific person's ask (docs' template omits it; the office one does not). */ +export function officeWakeMessage(docTitle: string, author: string, commentText: string): string { + return [ + `${author} asked you in a comment on the office document "${docTitle}":`, + ...commentText.split("\n").map((line) => `> ${line}`), + "You have been given edit access to this document; answer with the office_* tools.", + ].join("\n"); +} + +/** The mention grammar (plan 5.4, user decision): + * - literal "@agent" token -> the doc's OWNER home chat (meta.owner); ownerless docs: logged + skipped. + * - "@project:<name>" and any other bare "@<name>" -> case-insensitive EXACT display-name match among folders of kind "project"|"agent". The contract supersedes the plan's narrower split ("@project:" -> projects, bare -> agents): both forms search both kinds; names are non-whitespace runs, so folder names containing spaces cannot be mentioned in v1 (documented limitation). + * Unresolvable names and DUPLICATE names (two folders sharing a display name, e.g. agent "Twin" and project "Twin") get no wake and a log line -- no silent disambiguation (plan risk item). */ +const MENTION_RE = /(?<![\w.])(?:@(project:(\S+)|agent(?![\w-])|([A-Za-z0-9_][\w-]*)))/g; + +type Mention = { owner: boolean; name: string | null }; + +function parseMentions(text: string): Mention[] { + const out: Mention[] = []; + for (const m of text.matchAll(MENTION_RE)) { + if (m[1] === "agent") out.push({ owner: true, name: null }); + else if (m[2] !== undefined) out.push({ owner: false, name: m[2].replace(/[.,;:!?)\]}"']+$/, "") }); // trailing sentence punctuation is not part of the folder name + else out.push({ owner: false, name: m[3] }); + } + return out; +} + +function statePath(rootDir: string, docId: string): string { + return path.join(rootDir, "state", `${docId}.json`); +} + +/** Last-seen comment ids. Missing file -> null (first save). Undecodable or bad shape -> empty set with a log line (module header). */ +function readSeen(rootDir: string, docId: string): { ids: Set<string>; first: boolean } | null { + try { + const raw: unknown = JSON.parse(fs.readFileSync(statePath(rootDir, docId), "utf8")); + if (typeof raw !== "object" || raw === null || !Array.isArray((raw as Record<string, unknown>).ids)) throw new Error("bad shape"); + return { ids: new Set((raw as { ids: unknown[] }).ids.filter((x) => typeof x === "string")), first: false }; + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") return null; + logLine(`office ping: state file ${statePath(rootDir, docId)} is corrupt -- treating the last-seen set as empty`); + return { ids: new Set(), first: false }; + } +} + +function writeSeen(rootDir: string, docId: string, ids: Iterable<string>): void { + const p = statePath(rootDir, docId); + fs.mkdirSync(path.dirname(p), { recursive: true }); + const tmp = path.join(path.dirname(p), `.${path.basename(p)}.${process.pid}.tmp`); + fs.writeFileSync(tmp, JSON.stringify({ ids: [...new Set(ids)] }, null, 2)); + fs.renameSync(tmp, p); +} + +/** The store's index writer (its ONE path for index.json -- fresh read, dedupe by (chatId, commentId), atomic + * temp+rename): plan 5.4 requires the grant persisted BEFORE the bell rings (the woken pi's office_* tools read + * docsForHome, which reads index.json). This wrapper keeps this module's log lines: a vanished doc is logged, + * a duplicate adds nothing; "added"/"duplicate" are both fine to ring the bell for. */ +function addGrant(rootDir: string, docId: string, chatId: string, commentId: string, at: number): void { + const outcome = storeAddGrant(docId, chatId, commentId, at, rootDir); + if (outcome === null) logLine(`office ping: document ${docId} vanished from index between resolve and grant -- skipped`); +} + +/** Resolve one mention to a home chat id, or null (reason logged). */ +function resolveMention(mention: Mention, metaOwner: string | null): string | null { + if (mention.owner) { + if (!metaOwner) { + logLine("office ping: @agent in a comment on an ownerless document -- no wake"); + return null; + } + return metaOwner; // plan 2.1: meta.owner IS the creating agent/project's home chat id + } + const name = mention.name; + if (!name) return null; + const matches = folders().filter((f) => (f.kind === "project" || f.kind === "agent") && f.name.toLowerCase() === name!.toLowerCase()); + if (matches.length === 0) { + logLine(`office ping: @${name} in a comment matches no agent or project -- no wake`); + return null; + } + if (matches.length > 1) { + logLine(`office ping: @${name} is ambiguous (${matches.map((f) => `${f.kind}:${f.name}`).join(", ")}) -- no silent disambiguation, no wake`); + return null; + } + // Home chat per plan 2.1 (and the tools contract's own resolution): a folder's workspace path is exactly what + // chats.folder(id) resolves to, so the sessions working in that folder (chats.inFolder) are its home chats. + // Deterministic pick: lowest id (the extension queues wakes per chat; any session of the home answers). + const chats = inFolder(matches[0]!.path).sort((a, b) => a.id.localeCompare(b.id)); + if (chats.length === 0) { + logLine(`office ping: folder "${matches[0]!.name}" has no sessions to wake -- no wake`); + return null; + } + return chats[0]!.id; +} + +function processComment(docId: string, title: string, owner: string | null, c: Comment): void { + const mentions = parseMentions(c.text); + if (mentions.length === 0) return; + const at = Date.now(); + // Dedupe per comment AND per pass: one audit/grant/queue/tell per target chat, however often it is mentioned. + for (const chatId of new Set(mentions.map((m) => resolveMention(m, owner)).filter((x): x is string => x !== null))) { + addGrant(root(), docId, chatId, c.id, at); + record({ subject: title, reason: `comment mention ${c.id} by ${c.author}: access to this document granted and home woken`, chatId }); + const queue = wakes.get(chatId) ?? []; + if (!queue.some((e) => e.details.docId === docId && e.details.commentId === c.id)) { + queue.push({ text: officeWakeMessage(title, c.author, c.text), details: { docId, commentId: c.id } }); + wakes.set(chatId, queue); + } + const told = (cfg.tell ?? ((id: string, data: unknown) => portal().chats.tell(id, data)))(chatId, { type: "ping" }); + if (told && typeof (told as Promise<boolean>).catch === "function") { + (told as Promise<boolean>).catch((e: unknown) => logLine(`office ping: tell ${chatId} failed: ${(e as Error)?.message ?? e}`)); + } + } +} + +/** The registered onPersisted handler. Exported so unit tests call it directly without the real callback machinery. */ +export function handlePersisted(docId: string, buffer: Buffer): void { + try { + const rootDir = cfg.root ?? portal().dataDir(); + const meta = getDoc(docId, rootDir); + if (!meta) return; // deleted between fetch and persist -- nothing to diff + if (meta.ext !== "docx") return; // v1: docx only, xlsx comments deferred to M4 + const { comments } = parseDocx(buffer); + const prev = readSeen(rootDir, docId); + if (prev === null && meta.imported) { // imported with its own comment history: baseline what importFile recorded at IMPORT time -- + // an id NOT in that set appeared AFTER the file was imported (e.g. a mention added before the first save): it is new to + // this library and gets processed so its wake is not lost; only if processing succeeds does it become seen (retry rule as + // below). When the field is absent (unparseable source) we fall back to baselining everything this save holds. + const known = meta.importedBaseline !== undefined; + const preExisting = new Set(known ? meta.importedBaseline : comments.map((c) => c.id)); + const seen = new Set(preExisting); + for (const c of comments) { + if (!known || preExisting.has(c.id)) continue; // fallback or imported history: baselined, never woken + try { + processComment(docId, meta.title, meta.owner, c); + seen.add(c.id); + } catch (err) { + logLine("office ping: comment " + c.id + " of " + docId + " could not be processed and stays unseen for retry: " + ((err as Error)?.message ?? err)); + } + } + writeSeen(rootDir, docId, [...seen]); + return; + } + // Write the seen set AFTER processing and only with ids handled successfully: a comment whose wake failed + // stays unseen and is re-reported by diffComments on the next persist (retry), while one later failure must + // not leave earlier comments permanently unseen either. + // prev === null here means a created doc on its very first save (imported docs returned above): the seen set + // starts EMPTY, so every current comment -- including mentions typed during the first editing session -- is processed. + const added = diffComments([...(prev ? prev.ids : [])].map((id): Comment => ({ id, author: "", text: "" })), comments).added; + const seen = new Set(prev ? prev.ids : []); + for (const c of added) { + try { + processComment(docId, meta.title, meta.owner, c); + seen.add(c.id); + } catch (err) { + logLine("office ping: comment " + c.id + " of " + docId + " could not be processed and stays unseen for retry: " + ((err as Error)?.message ?? err)); + } + } + writeSeen(rootDir, docId, [...seen]); + } catch (err) { + logLine(`office ping: persist of ${docId} failed: ${(err as Error)?.message ?? err}`); + } +} diff --git a/extensions/office/src/portal.tsx b/extensions/office/src/portal.tsx new file mode 100644 index 000000000..3e332c5d7 --- /dev/null +++ b/extensions/office/src/portal.tsx @@ -0,0 +1,15 @@ +import type { Portal } from "@pithagoras/extension-api"; + +/** The portal this extension is loaded into, for every part of it: set before anything is registered. */ +export let portal: Portal; + +export function setPortal(given: Portal): void { + portal = given; +} + +// A text in the language in effect: said as written until a translation is added (the browser half ships English only in v1). +export const t = (text: string, vars?: Record<string, string | number>): string => portal.t(text, vars); + +/** The Office page -- and where it opens one document. */ +export const officeUrl = (docId?: string): string => + `/x/${portal.extension}/office${docId ? `?doc=${encodeURIComponent(docId)}` : ""}`; diff --git a/extensions/office/src/server.ts b/extensions/office/src/server.ts new file mode 100644 index 000000000..0036c11d0 --- /dev/null +++ b/extensions/office/src/server.ts @@ -0,0 +1,699 @@ +/** + * The server half of pithagoras-office -- the portal start/stop entry (the built server.js behind package.json's + * `server` field). Boots exactly what the contract's "server.ts" section fixes, over the seams this wave commits: + * + * 1. setPortal(portal) + the shared HS256 secret file dataDir()/jwt-secret (0600; created ONCE with + * crypto.randomBytes(32).toString("hex"), reused on every later boot -- one secret for callback JWTs, + * document tickets AND editor-config signing: plan 5.6). + * 2. registerTools({record}), wirePing({root, record}) -- both audit sinks land in portal().audit AND a log line. + * 3. The co-editor model (plan 5.7): there is NO co-editing switch and NO hook to wire -- office_edit + * proceeds under the strict mtime/revision gate no matter who has the editor open, and a stale edit is + * refused loudly as a conflict (fail loud, re-read + retry). Nothing here installs any refusal machinery. + * 4. callback.configure(...) once, with getState = () => container.readDsRecord(root) and the boot secret; + * live.setMetaResolver(docId -> store.getDoc(docId, root)?.mtimeMs ?? null). + * 5. The UI routes on portal.router (mounted by core at /api/x/pithagoras-office, session-gated there): see the + * route table in .reports/T-WIRE.md; every error is JSON {error, message?, code?}. + * 6. The cookie-free LOOPBACK listener (plan 5.2 design decision): http bound to 127.0.0.1 port 0 that serves ONLY + * GET /doc/<ticket> (a fresh document fetch for the editor: verifyTicket -> bytes, no-store) and POST /callback + * (the DS callback, callback.handleCallback). Everything else is a 404 JSON. This origin -- not dsOrigin() -- + * is what document.url and editorConfig.callbackUrl point at in the minted config (a container cannot fetch a + * browser-only origin; "container:<id>" origins have no URL at all). Divergence from the contract's one-liner, + * documented. + * 7. Boot: settleKept() (.trash >7d sweep + index-vs-disk prune) FIRST, then the workspace scan + * (store.scanWorkspace over portal().folders.list(): every .docx/.xlsx of the agent homes and projects + * indexed IN PLACE -- manual import is gone, the scanned file IS the document; POST /documents/rescan + * re-runs it on demand) + container.lookAfter() (adopt an existing DS), + * then a 30_000 ms unref'd interval that re-runs settleKept+lookAfter. The returned cleanup closes the listener and clears it. + * 8. deps.js, the pithagoras-docs pattern: ensureDependencies runs at boot IN THE BACKGROUND (this module has NO + * static jszip import, so a folder without node_modules imports cleanly) and arms jsZip() to wait for it -- + * the first template call blocks on the install; later calls use the one cached load. A create that still + * hits an unready dependency answers 503 code "dependencies_pending" (retryable) and logs why. + * + * dsOrigin(): ONE function, per the contract -- the origin read straight out of ds.json (network-mode option (a), + * plan 5.2: host-networked DS at http://127.0.0.1). The BROWSER now reaches it through the portal's GET /ds/ reverse + * proxy (api.js, the editor iframe and every asset -- same-origin, behind the session gate like every route here): + * apiJs is that root-relative prefix, and api.js' getBasePath() turns it into the editor's base URL. What stays an + * ABSOLUTE LOOPBACK URL in the signed config is document.url and editorConfig.callbackUrl: those are fetched by the + * DS server-side (host netns -> loopback works), never by a browser -- leave them exactly as-is. When the recorded + * origin is a "container:<id>" marker there is NO usable URL in v1: it is returned as-is and /editor-config AND /ds + * refuse with a clear JSON error saying that deployment shape needs an M0 decision. + */ +import http from "node:http"; +import type { AddressInfo } from "node:net"; +import os from "node:os"; +import fs from "node:fs"; +import path from "node:path"; +import { randomBytes } from "node:crypto"; +import type { ServerPortal } from "@pithagoras/extension-api/server"; +import { setPortal, portal, dataDir } from "./host"; +import * as container from "./container"; +import { loadSettings, saveSettings, type OfficeSettings } from "./settings"; +import { signJwt } from "./jwt"; +import { mintTicket, verifyTicket } from "./tickets"; +import * as store from "./store"; +import * as live from "./live"; +import * as callback from "./callback"; +import { wirePing } from "./ping"; +import { registerTools } from "./tools"; +import { ensureDependencies } from "../deps.js"; // the pithagoras-docs pattern: its own node_modules installed at boot, in the background +import { armJsZipInstall } from "./jszip"; + +/* ------------------------------------------------------------------ */ +/* constants */ +/* ------------------------------------------------------------------ */ + +/** The editor-config JWT lives 15 minutes: the official DS signing sample uses expiresIn 15*60, and a config is + * only ever handed to a fresh api.js load right after minting. */ +const CONFIG_TTL_MS = 15 * 60 * 1000; +/** A document ticket lives 10 minutes: one editor open-and-fetch window (the tickets module's own default). */ +const TICKET_TTL_MS = 600_000; +/** The fallback identity the minted config claims: portal exposes no per-route session user in v1, so every + * co-edited doc shows this seat (plan 5.6 editorConfig.user). */ +const EDITOR_USER = { id: "portal-user", name: "Portal" } as const; + +const DOCX_MEDIA = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; +const XLSX_MEDIA = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; +function mediaType(ext: "docx" | "xlsx"): string { + return ext === "docx" ? DOCX_MEDIA : XLSX_MEDIA; +} + +/* ------------------------------------------------------------------ */ +/* module state (one boot per portal start; the host re-runs us) */ +/* ------------------------------------------------------------------ */ + +/** Port of the loopback listener once booted (0 before boot / after cleanup); tests reach it via this. */ +let loopPort = 0; +export function loopbackPort(): number { + return loopPort; +} + +function logLine(line: string): void { + try { + portal().log(line); + } catch { + // pre-portal (unit seams) or a dead portal -- console is the floor, never crash a boot. + console.error(line); + } +} + +/** The ONE audit sink for this wave: every mutating op lands in portal().audit AND as one log line. */ +function recordAudit(entry: { subject: string; reason?: string; chatId?: string }): void { + const line = `pithagoras-office: ${entry.subject}${entry.reason ? ` -- ${entry.reason}` : ""}${entry.chatId ? ` (chat ${entry.chatId})` : ""}`; + try { + portal().audit.record({ subject: entry.subject, reason: entry.reason, chatId: entry.chatId }); + } catch (e) { + logLine(`audit.record refused the entry: ${(e as Error).message}`); + } + logLine(line); +} + +/* ------------------------------------------------------------------ */ +/* the shared secret */ +/* ------------------------------------------------------------------ */ + +/** Create dataDir()/jwt-secret ONCE -- mode 0600, explicit chmod (umask-proof), hex of crypto.randomBytes(32) -- and + * return its trimmed content. Existing files are reused byte-for-byte (second boot and every one after). */ +function ensureJwtSecret(dd: string): string { + fs.mkdirSync(dd, { recursive: true }); + const file = path.join(dd, "jwt-secret"); + let raw = ""; + try { + raw = fs.readFileSync(file, "utf8"); + } catch { + raw = ""; // missing: generated below + } + if (raw.trim().length < 32) fs.writeFileSync(file, randomBytes(32).toString("hex") + "\n", { mode: 0o600 }); // empty or truncated too soon to be a usable secret: (re)generate rather than reuse + fs.chmodSync(file, 0o600); // the exact perms either way -- never rely on the process umask + return fs.readFileSync(file, "utf8").trim(); +} + +/** Read the boot secret from `root` (the portal dataDir by default). Boot creates the file; this only reads. */ +export function jwtSecret(root?: string): string { + return fs.readFileSync(path.join(root ?? dataDir(), "jwt-secret"), "utf8").trim(); +} + +/* ------------------------------------------------------------------ */ +/* settleKept */ +/* ------------------------------------------------------------------ */ + +/** Reconcile the library with what is on disk: sweep .trash entries older than 7 days, then drop index entries + * whose library file vanished (store.pruneMissingDocs). The index is the source of truth -- a docs/<id>/ dir with no + * entry is NOT added back. Pure function of (root): booted at start and on the interval, unit-tested directly. */ +export function settleKept(root?: string): { sweptTrash: number; prunedDocs: string[] } { + const dd = root ?? dataDir(); + return { sweptTrash: store.sweepTrash(dd), prunedDocs: store.pruneMissingDocs(dd) }; +} + +/* ------------------------------------------------------------------ */ +/* small response helpers */ +/* ------------------------------------------------------------------ */ + +/** A failure of the LAZY jszip load (deps.js): its module import is refused while npm still installs in the + * background at boot, or the install failed and the next call has not retried yet. Recognized on the error chain, + * not by message cosmetics alone: ERR_MODULE_NOT_FOUND naming jszip, or a message that names the dependency + * install. A 503 retryable answer is right for BOTH (the other codes -- exists/corrupt index -- are not). */ +function isDependencyError(e: unknown): boolean { + let cur: unknown = e; + for (let i = 0; cur && i < 10; i++) { + const ce = cur as { code?: string; message?: string }; + if (ce.code === "ERR_MODULE_NOT_FOUND" && /jszip/i.test(ce.message ?? "")) return true; + if (/Cannot find package 'jszip'|installing its dependencies|Could not install its dependencies/i.test(ce.message ?? "")) return true; + cur = (cur as { cause?: unknown }).cause; + } + return false; +} + +const CODE_STATUS: Record<string, number> = { + not_found: 404, + exists: 409, + unsupported_format: 415, + invalid_title: 400, + revision_mismatch: 409, + index_corrupt: 500, + no_docker: 503, + too_large: 413, // store.importFile's 512 MiB import cap +}; + +/** Map the coded errors store/container throw onto consistent JSON {error, code?}. */ +function codedResponse(res: { status: (c: number) => { json: (b: unknown) => void } }, e: unknown): void { + const ce = e as Error & { code?: string }; + res.status(CODE_STATUS[ce.code ?? ""] ?? 500).json({ error: ce.message, ...(ce.code ? { code: ce.code } : {}) }); +} + +/** Where a doc's bytes live NOW: a scanned doc's real workspace file (DocMeta.path), else its library folder. */ +function fileFor(root: string, doc: store.DocMeta): string { + return doc.path ?? path.join(root, "docs", doc.docId, `${doc.title}.${doc.ext}`); +} + +/** Content-Disposition for a download: an ASCII fallback name plus the RFC 5987 UTF-8 form for any non-ASCII title. */ +function attachmentHeader(title: string, ext: string): string { + const raw = `${title}.${ext}`; + const ascii = raw.replace(/[^\x20-\x7e]/g, "-").replace(/["\\;:,]/g, "'"); + return `attachment; filename="${ascii}"; filename*=UTF-8''${encodeURIComponent(raw)}`; +} + +function jsonRaw(res: http.ServerResponse, status: number, body: unknown): void { + if (res.headersSent) return; + res.writeHead(status, { "Content-Type": "application/json" }); + res.end(JSON.stringify(body)); +} + +/* ------------------------------------------------------------------ */ +/* the loopback listener (plan 5.2) */ +/* ------------------------------------------------------------------ */ + +function makeLoopbackHandler(dd: string, secret: string): (req: http.IncomingMessage, res: http.ServerResponse) => void { + return (req, res) => { + let pathname: string; + try { + pathname = new URL(req.url ?? "/", "http://127.0.0.1").pathname; + } catch { + jsonRaw(res, 404, { error: "no such endpoint on the office loopback listener", code: "not_found" }); + return; + } + if (req.method === "GET" && pathname.startsWith("/doc/")) { + // Tickets are base64url tokens: no percent-decoding -- decodeURIComponent would throw URIError on a + // malformed escape, and an uncaught throw in the http listener kills the server. verifyTicket enforces the alphabet anyway. + const ticket = pathname.slice("/doc/".length); + const verified = verifyTicket(ticket, secret); + if (!verified) { + jsonRaw(res, 401, { error: "invalid or expired document ticket", code: "bad_ticket" }); + return; + } + const doc = store.getDoc(verified.docId, dd); + if (!doc) { + jsonRaw(res, 404, { error: `no such document ${verified.docId}`, code: "not_found" }); + return; + } + let bytes: Buffer; + try { + bytes = fs.readFileSync(fileFor(dd, doc)); + } catch { + jsonRaw(res, 404, { error: `the library file for ${doc.docId} is missing from disk`, code: "not_found" }); + return; + } + res.writeHead(200, { "Content-Type": mediaType(doc.ext), "Cache-Control": "no-store" }); + res.end(bytes); + return; + } + if (req.method === "POST" && pathname === "/callback") { + callback.handleCallback(req, res); + return; + } + jsonRaw(res, 404, { error: `no such endpoint on the office loopback listener (${req.method} ${pathname})`, code: "not_found" }); + }; +} + +/* ------------------------------------------------------------------ */ +/* editor-config */ +/* ------------------------------------------------------------------ */ + +/** Build (or refuse) the new DocsAPI.DocEditor config for `docId`. The plain object is signed WHOLE -- per the DS + * signing rule ("the payload must have the same structure as the config"; since 7.1 sign exactly what you send) -- + * and returned as {...config, token, apiJs}: the page drops `apiJs` before calling DocEditor, everything else goes in. */ +function buildEditorConfig(dd: string, secret: string, docId: string): { res?: unknown; config?: Record<string, unknown>; ticketUrl?: string } { + const doc = store.getDoc(docId, dd); + if (!doc) return { res: { status: 404, body: { error: `no document with id ${docId}`, code: "not_found" } } }; + + const record = container.readDsRecord(dd); + if (!record) return { res: { status: 503, body: { error: "the document server is not configured yet -- start it from the Office page first", code: "ds_not_configured" } } }; + // network-mode option (a): a "container:<id>" origin is a marker with no browser-reachable URL in v1 -- fail clearly. + if (record.origin.startsWith("container:")) { + return { res: { status: 422, body: { error: `the document server runs under ${record.origin}: that deployment shape cannot expose the editor to a browser until the M0 spike-2 decision lands`, code: "unresolvable_ds_origin", origin: record.origin } } }; + } + + const { token: ticket } = mintTicket(doc.docId, TICKET_TTL_MS, secret); + // plan 5.2: document.url is the ticketed GET on OUR loopback listener (the container's fetch must reach a URL it + // can actually dial -- never a browser-only origin); callbackUrl sits right next door. + const base = `http://127.0.0.1:${loopPort}`; + const config: Record<string, unknown> = { + documentType: doc.ext === "docx" ? "word" : "cell", + type: "embedded", + title: doc.title, + document: { + fileType: doc.ext, + key: callback.makeKey(doc.docId, doc.mtimeMs), + title: `${doc.title}.${doc.ext}`, + url: `${base}/doc/${ticket}`, + }, + editorConfig: { + callbackUrl: `${base}/callback`, + mode: "edit", + lang: "en", // v1 default -- portal exposes no per-user locale seam + user: { id: EDITOR_USER.id, name: EDITOR_USER.name }, + }, + }; + live.setUser(doc.docId, { id: EDITOR_USER.id, name: EDITOR_USER.name }, true); // the seat opens: presence on mint + return { config, ticketUrl: `${base}/doc/${ticket}` }; +} + +/* ------------------------------------------------------------------ */ +/* the UI routes (core mounts portal.router at /api/x/pithagoras-office)*/ +/* ------------------------------------------------------------------ */ + +function registerRoutes(dd: string, secret: string): void { + const r = portal().router; + const json = portal().http.json(); + + /** GET /state -> settings + container status + the document list + who has what open. The poller's payload. */ + r.get("/state", async (_req, res) => { + const s = loadSettings(dd); + let c: Awaited<ReturnType<typeof container.status>>; + try { + c = await container.status(undefined, dd); + } catch (e) { + c = { state: "stopped", pull: null } as typeof c; // status() should not throw -- never kill the poller + logLine(`container.status failed: ${(e as Error).message}`); + } + const docs = store.listDocs(dd); + const locks: Record<string, { id: string; name?: string }[]> = {}; + for (const d of docs) { + // every doc gets an entry -- the client badge code reads locks[docId] without a null check + locks[d.docId] = live.usersOf(d.docId).map((u) => ({ id: u.id, name: u.name })); + } + res.json({ enabled: s.enabled, imageTag: s.imageTag, container: c, docs, locks }); + }); + + /** POST /documents {title, type:"word"|"cell", owner?} -> 201 meta; 409 exists. A jszip load failure is NOT the + * ordinary coded response: the dependency may still be installing in the background at boot (deps.js), so the + * answer is a retryable 503 with code "dependencies_pending" -- the UI shows it verbatim and the click never dies + * silently. Every other failure is logged too, so a live failure finally LEAVES A LINE in the portal's log + * (the pre-fix routes logged nothing, which is why the first live create-failure had no trace to read). */ + r.post("/documents", json, async (req, res) => { + const body = (req.body ?? {}) as Record<string, unknown>; + const title = typeof body.title === "string" ? body.title : ""; + const type = body.type; + if (title.trim() === "") return res.status(400).json({ error: "a document needs a non-empty { title }", code: "invalid_title" }); + if (type !== "word" && type !== "cell") return res.status(400).json({ error: '{ type } must be "word" or "cell"', code: "unsupported_format" }); + const owner = typeof body.owner === "string" && body.owner !== "" ? body.owner : null; + try { + const doc = await store.createDoc({ title, type, owner }, dd); + recordAudit({ subject: title, reason: `created ${doc.ext} document ${doc.docId}`, chatId: owner ?? undefined }); + res.status(201).json(doc); + } catch (e) { + if (isDependencyError(e)) { + logLine(`POST /documents failed while its jszip dependency is not ready yet: ${(e as Error).message}`); + return res.status(503).json({ + error: "the document templates are not ready yet -- the extension's jszip dependency is still installing in the background (or its install failed; see the portal log). Retry in a few seconds.", + code: "dependencies_pending", + }); + } + logLine(`POST /documents failed: ${(e as Error).message}`); + codedResponse(res, e); + } + }); + + /** POST /documents/rescan -> { added, updated }: re-run the workspace scan NOW over every agent folder (fresh from + * portal().folders.list()), in addition to the automatic one at boot. The library UI's Rescan button calls it. */ + r.post("/documents/rescan", json, async (_req, res) => { + try { + const out = store.scanWorkspace(portal().folders.list().map((f) => f.path), dd); + logLine(`workspace rescan: ${out.added} added, ${out.updated} refreshed`); + res.json(out); + } catch (e) { + logLine(`workspace rescan failed: ${(e as Error).message}`); + codedResponse(res, e); + } + }); + + /** POST /documents/:id/rename {title} -> 200 the new meta; keeps mtimeMs (the open sessions' revision survives a retitle). */ + r.post("/documents/:id/rename", json, async (req, res) => { + const id = String(req.params?.id ?? ""); + const body = (req.body ?? {}) as Record<string, unknown>; + const title = typeof body.title === "string" ? body.title : ""; + if (title.trim() === "") return res.status(400).json({ error: "a rename needs a non-empty { title }", code: "invalid_title" }); + try { + const doc = store.renameDoc(id, title, dd); + recordAudit({ subject: `${doc.title} (${id})`, reason: `renamed from ${doc.title}`, chatId: undefined }); + res.json(doc); + } catch (e) { + codedResponse(res, e); + } + }); + + /** GET /documents/:id -> the library bytes as an attachment (download). */ + r.get("/documents/:id", async (req, res) => { + const doc = store.getDoc(String(req.params?.id ?? ""), dd); + if (!doc) return res.status(404).json({ error: `no document with id ${String(req.params?.id ?? "")}`, code: "not_found" }); + let bytes: Buffer; + try { + bytes = fs.readFileSync(fileFor(dd, doc)); + } catch { + return res.status(404).json({ error: `the library file for ${doc.docId} is missing from disk`, code: "not_found" }); + } + res.setHeader("Content-Type", mediaType(doc.ext)); + res.setHeader("Cache-Control", "no-store"); + res.setHeader("Content-Disposition", attachmentHeader(doc.title, doc.ext)); + res.end(bytes); + }); + + /** DELETE /documents/:id -> moved to .trash (settleKept reclaims after 7 days) + live presence cleared. */ + r.delete("/documents/:id", async (req, res) => { + const id = String(req.params?.id ?? ""); + try { + store.deleteDoc(id, dd); + } catch (e) { + codedResponse(res, e); + return; + } + live.clearDoc(id); // nobody "has it open" for a gone doc anymore + recordAudit({ subject: id, reason: "deleted (moved to .trash)" }); + res.json({ ok: true }); + }); + + /** GET /documents/:id/export -> a FRESH copy of the current bytes, exported to a temp dir, served, and the temp + * gone again (store.exportDoc writes `${title}.ext` next to nothing else -- nothing is ever overwritten). */ + r.get("/documents/:id/export", async (req, res) => { + const id = String(req.params?.id ?? ""); + const doc = store.getDoc(id, dd); + if (!doc) return res.status(404).json({ error: `no document with id ${id}`, code: "not_found" }); + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "office-export-")); + let bytes: Buffer; + try { + const written = store.exportDoc(id, tmp, dd); + bytes = fs.readFileSync(written); + recordAudit({ subject: doc.title, reason: `exported (download of a fresh copy)` }); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + res.setHeader("Content-Type", mediaType(doc.ext)); + res.setHeader("Cache-Control", "no-store"); + res.setHeader("Content-Disposition", attachmentHeader(doc.title, doc.ext)); + res.end(bytes); + }); + + /** POST /editor-config {doc} -> the config object for `new DocsAPI.DocEditor(el, cfg)`: the plain fields plus + * token = signJwt(the WHOLE plain config minus the token itself, 15 min) and apiJs (the script src -- drop it + * before calling DocEditor). On a mint the seat joins live presence. See buildEditorConfig for the refusals. */ + r.post("/editor-config", json, async (req, res) => { + const body = (req.body ?? {}) as Record<string, unknown>; + const docId = typeof body.doc === "string" && body.doc !== "" ? body.doc : ""; + if (!docId) return res.status(400).json({ error: 'the body needs { doc: "<document id>" }' }); + const built = buildEditorConfig(dd, secret, docId); + if (built.res) { + const rr = built.res as { status: number; body: Record<string, unknown> }; + return res.status(rr.status).json(rr.body); + } + const config = built.config as Record<string, unknown>; + // purpose "editor-config" (S3): the signed typ claim tags this token's family for any future in-portal + // verification; DS itself checks only the HMAC/expiry and ignores the extra claim. + // apiJs: ROOT-RELATIVE through this extension's /ds proxy. The browser resolves it against its own page origin + // (exactly where the user reached the portal, session-gated) -- and api.js' getBasePath() takes everything before + // "api/documents/api.js" in its script src as the editor iframe base, so the whole editor runs under that prefix. + // It is NOT part of the signed payload (the client drops it before calling DocEditor). + res.json({ ...config, token: signJwt(config, CONFIG_TTL_MS, secret, "editor-config"), apiJs: `/api/x/${encodeURIComponent(portal().extension)}/ds/web-apps/apps/api/documents/api.js` }); + }); + + /** POST /settings {enabled?, imageTag?} -> saved + side effects: enabling fires ensureContainer WITHOUT + * (There is NO co-editing switch in this payload: the agent is a co-editor under the revision gate, full stop.) + * awaiting it forever (catch+log; the response says `starting:true` and reports the container state AS OF NOW -- + * a poller watches it arrive), disabling stops it now. :latest is refused -- a moved tag bricks co-editing keys. */ + r.post("/settings", json, async (req, res) => { + const body = (req.body ?? {}) as Record<string, unknown>; + const before = loadSettings(dd); + const patch: Partial<OfficeSettings> = {}; + if ("enabled" in body) { + if (typeof body.enabled !== "boolean") return res.status(400).json({ error: "{ enabled } must be a boolean", code: "invalid_enabled" }); + patch.enabled = body.enabled as boolean; + } + if ("imageTag" in body) { + const tag = typeof body.imageTag === "string" ? body.imageTag.trim() : ""; + if (tag === "") return res.status(400).json({ error: "{ imageTag } must be a non-empty string like onlyoffice/documentserver:10.0", code: "invalid_image_tag" }); + if (/:(latest|stable)$/i.test(tag) || tag.endsWith("@latest")) return res.status(400).json({ error: "never pin :latest -- it moves under you and breaks co-editing keys", code: "invalid_image_tag" }); + patch.imageTag = tag; + } + if (Object.keys(patch).length === 0) return res.status(400).json({ error: "no known setting in the body -- send { enabled?, imageTag? }" }); + + const settings = saveSettings(patch, dd); + recordAudit({ subject: "pithagoras-office settings", reason: `enabled=${settings.enabled} imageTag=${settings.imageTag}` }); + + let starting = false; + if (patch.enabled === true && !before.enabled) { + // fire-and-forget on purpose: a first pull takes minutes and must not hold the response open + starting = true; + void container.ensureContainer(undefined, undefined, dd).then((r2) => recordAudit({ subject: container.DS_NAME, reason: r2.started ? "document server started (settings)" : "document server already running" })).catch((e) => logLine(`could not start the document server: ${(e as Error).message}`)); + } + if (patch.enabled === false && before.enabled) { + try { + await container.stopContainer(undefined, dd); // records the user-stopped intent either way + } catch (e) { + return res.status(503).json({ error: `settings were saved but the container could not be stopped: ${(e as Error).message}`, code: "no_docker", settings }); + } + } + + const c = await container.status(undefined, dd); + res.json({ settings, container: c, ...(starting ? { starting: true } : {}) }); + }); + + /** GET /container/status -> the same object status() builds: state (+ version/memBytes when up), and pull -- the + * latest pull line while a first start is downloading (the POST /container/start route awaits the whole ensure; + * progress surfaces here via polling). */ + r.get("/container/status", async (_req, res) => { + try { + res.json(await container.status(undefined, dd)); + } catch (e) { + codedResponse(res, e); + } + }); + + /** POST /container/start -> ensureContainer to completion: adopt or make+start the DS, ds.json written, and the + * response carries the resulting status. */ + r.post("/container/start", async (_req, res) => { + try { + const r2 = await container.ensureContainer(undefined, undefined, dd); + recordAudit({ subject: container.DS_NAME, reason: r2.started ? "document server started" : "document server already running" }); + res.json({ ...r2, container: await container.status(undefined, dd) }); + } catch (e) { + codedResponse(res, e); + } + }); + + /** POST /container/stop -> stop + record the user-stopped intent (lookAfter stays quiet while it stands). */ + r.post("/container/stop", async (_req, res) => { + try { + await container.stopContainer(undefined, dd); + recordAudit({ subject: container.DS_NAME, reason: "document server stopped" }); + res.json({ container: await container.status(undefined, dd) }); + } catch (e) { + codedResponse(res, e); + } + }); + + /** GET+POST /ds/<rest...> -> a thin reverse proxy for the browser's Document Server traffic (api.js, the editor + * iframe and its assets -- AND the socket.io POLLING channel). The editor app talks to docserver over engine.io: + * GET <app>/10.0.0-<buildhash>/doc/<key>/c/?EIO=4&transport=polling handshakes, then the client's packets (the + * "40" connect packet, user state, ...) arrive as POSTs to the same path while its next-poll GETs stay open until + * the server has something to send. A GET-only proxy answered every POST with the core's 404 and the channel never + * opened: the editor white-loads forever even though DS is healthy (reproduced by wire replay, see PR307 notes). + * Core mounts us at /api/x/<extension> BEHIND THE SESSION GATE (core/server/src/extension-host.ts), so only + * signed-in browsers may reach this -- and it forwards to ONE origin only: the recorded DS origin from ds.json, + * never anywhere else. The upstream path is recomputed from the RAW request URL with its query string included + * (express params may be decoded; an iframe's ?dockey=... must pass verbatim). The upstream ask carries + * accept-encoding: identity and neither content-encoding nor content-length is copied on the way out: + * Node's undici fetch AUTO-DECODES gzip bodies but keeps the upstream's compressed content-encoding and COMPRESSED + * content-length in the response headers, so copying them made Node emit Content-Length for 13.7KB while sending + * 67KB of decoded bytes -- browsers read a truncated api.js (invalid JS) and the keep-alive stream reset. Asking + * identity means the body arrives plain with a truthful length anyway; buffering it and letting Node compute + * Content-Length from the ACTUAL bytes makes header and payload exactly consistent. + * Two transport rules beyond that: polling GETs are HELD open by docserver until it has data (an engine.io ping + * lands at least every 25s), so the abort window must cover a full ping cycle or the proxy kills healthy long- + * polls -- undici's abort also closes the upstream socket, which DS logs as a 499 at exactly the cutoff. And the + * handshake JSON advertises upgrades:["websocket"]: this proxy is fetch-based and cannot relay WebSocket frames, + * so "websocket" is stripped from every outbound handshake, pinning the client to polling for good (engine.io's + * fallback on a failed upgrade works, but a deterministic single transport has no double-connection churn). + * Refusals mirror /editor-config: 503 ds_not_configured (no ds.json), 422 unresolvable_ds_origin (a + * "container:<id>" marker has no URL until M0), 503 ds_unavailable (the origin did not answer). Nothing below + * this route mutates portal state: POSTs only ever forward to the DS. */ + const proxyDs = async (req: any, res: any): Promise<void> => { + const raw = typeof req.url === "string" ? req.url : ""; + const i = raw.indexOf("/ds/"); + if (i === -1) return res.status(404).json({ error: "bad proxy path", code: "not_found" }); + const rest = raw.slice(i + 4); // everything after /ds/, query string included, untouched + const origin = dsOrigin(dd); + if (origin === "") { + return res.status(503).json({ error: "the document server is not configured yet -- start it from the Office page first", code: "ds_not_configured" }); + } + if (origin.startsWith("container:")) { + return res.status(422).json({ error: `the document server runs under ${origin}: that deployment shape cannot expose the editor to a browser until the M0 spike-2 decision lands`, code: "unresolvable_ds_origin", origin }); + } + const fwdHeaders: Record<string, string> = { "accept-encoding": "identity" }; + let body: Buffer | undefined; + if (req.method !== "GET" && req.method !== "HEAD") { + // engine.io POSTs its client packets as raw text/plain; buffer them verbatim. Nothing upstream of this + // route parses non-JSON bodies, so the stream is still ours when it is one at all. + const chunks: Buffer[] = []; + let total = 0; + for await (const c of req) { + total += c.length; + if (total > 128 * 1024 * 1024) break; // the handshake advertises maxPayload 100MB; stay above it + chunks.push(c); + } + body = Buffer.concat(chunks); + const ct = req.headers?.["content-type"]; + if (typeof ct === "string") fwdHeaders["content-type"] = ct; + } + let upstream: Response; + try { + upstream = await fetch(`${origin}/${rest}`, { + method: req.method, + headers: fwdHeaders, + body: body && body.length > 0 ? new Uint8Array(body) : undefined, + redirect: "follow", + // An idle poll must survive a full engine.io ping cycle (pingInterval 25s) or more; 15s cut healthy + // long-polls in half and made every doc channel look like it was dying. + signal: AbortSignal.timeout(90_000), + }); + } catch (e) { + logLine(`${req.method} /ds/${rest.split("?")[0]} -> ${origin} failed: ${(e as Error).message}`); + return res.status(503).json({ error: `the document server at ${origin} did not answer`, code: "ds_unavailable" }); + } + // content-encoding / content-length are DELIBERATELY not copied: undici has already decoded any gzip body, + // so the upstream length now describes compressed bytes we never send. The buffer below IS the payload; + // Node computes a truthful Content-Length from it. + for (const k of ["content-type", "cache-control", "content-disposition", "access-control-allow-origin"]) { + const v = upstream.headers.get(k); + if (v !== null) res.setHeader(k, v); + } + // Everything else (hop-by-hop, set-cookie, server banners, ...) is dropped. + try { + let buf = Buffer.from(await upstream.arrayBuffer()); + // Pin the socket.io client to polling: strip "websocket" from the engine.io handshake's upgrades list -- + // this proxy cannot relay WebSocket frames (fetch-based), so a promised upgrade could only end in churn. + if (upstream.status === 200 && buf.length > 2 && buf[0] === 0x30 /* '0', the EIO handshake prefix */ && buf[1] === 0x7b /* '{' */ && !rest.includes("/a/") && !rest.includes("/u/") && !rest.includes("/s/")) { + try { + const h = JSON.parse(buf.subarray(1).toString("utf8")); // skip the '0' packet prefix + if (Array.isArray(h.upgrades) && h.upgrades.includes("websocket")) { + h.upgrades = h.upgrades.filter((t: unknown) => t !== "websocket"); + // the '0' EIO packet-type prefix is PART of the wire format -- re-serialize behind it + buf = Buffer.from("0" + JSON.stringify(h), "utf8"); + } + } catch { + /* not a handshake shape after all -- forward untouched */ + } + } + res.status(upstream.status).end(buf.length === 0 ? undefined : buf); + } catch (e) { + logLine(`${req.method} /ds/${rest.split("?")[0]}: the body from ${origin} could not be read: ${(e as Error).message}`); + if (!res.headersSent) res.status(503).json({ error: `the document server at ${origin} did not answer`, code: "ds_unavailable" }); + else res.destroy(); + } + }; + r.get(/^\/ds\/(.*)$/, proxyDs); + r.post(/^\/ds\/(.*)$/, proxyDs); +} + +/* ------------------------------------------------------------------ */ +/* boot + cleanup */ +/* ------------------------------------------------------------------ */ + +/** dsOrigin(): the browser-facing origin of the DS read from ds.json (option (a)). "" when unconfigured; a + * "container:<id>" marker is returned AS-IS -- it is the clear signal that /editor-config must refuse with the M0 note. */ +export function dsOrigin(root?: string): string { + return container.readDsRecord(root ?? dataDir())?.origin ?? ""; +} + +export default async function officeServer(portalObj: ServerPortal): Promise<() => Promise<void>> { + setPortal(portalObj); + // The pithagoras-docs pattern: a folder without its own node_modules gets them installed in the BACKGROUND -- + // boot does not wait (the portal starts the rest), and jsZip() awaits this promise before first use. + void armJsZipInstall(ensureDependencies(logLine)); + const dd = portal().dataDir(); + const secret = ensureJwtSecret(dd); // the one HS256 secret: callbacks, tickets and configs all sign with it (plan 5.6) + + // the seams, exactly as their module headers contract them (audit lands in portal().audit AND a log line). + // NOTE: there is NO co-editing switch and no flush-before-write hook to install here -- office_edit runs under + // the strict mtime/revision gate with human editor sessions open or not (co-editor model; fail loud, re-read + retry). + registerTools({ root: dd, record: recordAudit }); + wirePing({ root: dd, record: recordAudit }); + callback.configure({ getState: () => container.readDsRecord(dd), getSecret: () => secret, root: dd }); + live.setMetaResolver((docId) => store.getDoc(docId, dd)?.mtimeMs ?? null); + + registerRoutes(dd, secret); + + // the cookie-free loopback listener (plan 5.2): 127.0.0.1, port 0 -> a private ephemeral port per boot + const server = http.createServer(makeLoopbackHandler(dd, secret)); + await new Promise<void>((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", () => resolve()); + }); + loopPort = (server.address() as AddressInfo).port; + + // boot settle: reclaim trash + reconcile index vs disk, then adopt an existing DS if settings say it should run + try { + settleKept(dd); + } catch (e) { + logLine(`settleKept at boot failed: ${(e as Error).message}`); + } + // the workspace scan indexes every .docx/.xlsx of the agent folders IN PLACE (product decision: no manual import + // any more -- what the agents work in IS the library). settleKept runs FIRST on purpose: it prunes entries whose + // file vanished while the portal slept, so the scan then REFRESHES only surviving files instead of re-adding them. + try { + const out = store.scanWorkspace(portal().folders.list().map((f) => f.path), dd); + logLine(`workspace scan at boot: ${out.added} added, ${out.updated} refreshed`); + } catch (e) { + logLine(`workspace scan at boot failed: ${(e as Error).message}`); + } + container.lookAfter(undefined, dd); // fire-and-forget by contract; logs its own failures + + let stopped = false; + const timer = setInterval(() => { + if (stopped) return; + try { + settleKept(dd); + } catch (e) { + logLine(`settleKept failed: ${(e as Error).message}`); + } + container.lookAfter(undefined, dd); + }, 30_000); + timer.unref(); // never hold the portal process open + + return async () => { + stopped = true; + clearInterval(timer); + loopPort = 0; + // close() alone would wait on keep-alive connections held open by clients (undici fetch among them); + // dropping the idle ones is what lets a portal stop -- and a test's cleanup -- actually finish + server.closeAllConnections(); + await new Promise<void>((resolve) => server.close(() => resolve())); + }; +} diff --git a/extensions/office/src/settings.ts b/extensions/office/src/settings.ts new file mode 100644 index 000000000..7495e4fb2 --- /dev/null +++ b/extensions/office/src/settings.ts @@ -0,0 +1,58 @@ +/** + * pithagoras-office settings: dataDir()/settings.json, merged over defaults field by field. + * + * Pure over a root directory -- no portal calls happen here except through the `root` default + * (dataDir()), and no logging: a missing or corrupt file yields the defaults SILENTLY. The server + * logs at its call site if it cares; keeping this module side-effect free also lets unit tests pass + * plain tmp dirs without reaching portal(). + */ +import fs from "node:fs"; +import path from "node:path"; +import { dataDir } from "./host"; + +export type OfficeSettings = { enabled: boolean; imageTag: string }; + +/** Pinned image tag -- never :latest (the settings page warns about it). */ +export const DEFAULT_SETTINGS: OfficeSettings = { enabled: false, imageTag: "onlyoffice/documentserver:10.0" }; + +const FILE = "settings.json"; + +function field<K extends keyof OfficeSettings>(raw: Record<string, unknown>, key: K): OfficeSettings[K] { + const value = raw[key]; + return value !== undefined && typeof value === typeof DEFAULT_SETTINGS[key] ? (value as OfficeSettings[K]) : DEFAULT_SETTINGS[key]; +} + +/** Read root/settings.json and merge it over DEFAULTS field by field; unknown fields are ignored -- LEGACY files + * written before the co-editing switch existed still carry a "coEdit" key, which is loaded without crashing and + * simply never surfaced. Missing file, unreadable file, or anything that is not a JSON object -> DEFAULT_SETTINGS + * (returned silently -- no logging in this module). */ +export function loadSettings(root = dataDir()): OfficeSettings { + let raw: unknown; + try { + raw = JSON.parse(fs.readFileSync(path.join(root, FILE), "utf8")); + } catch { + return { ...DEFAULT_SETTINGS }; + } + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) return { ...DEFAULT_SETTINGS }; + const obj = raw as Record<string, unknown>; + return { enabled: field(obj, "enabled"), imageTag: field(obj, "imageTag") }; +} + +/** Load, apply a patch of any subset of fields (ignoring absent/undefined ones), persist the whole object atomically + * (tmp file in the same directory + rename) and return the stored result. */ +export function saveSettings(patch: Partial<OfficeSettings>, root = dataDir()): OfficeSettings { + const merged: OfficeSettings = { ...loadSettings(root) }; + for (const [key, value] of Object.entries(patch)) { + if (value === undefined || !(key in DEFAULT_SETTINGS)) continue; // unknown keys are ignored + (merged as Record<string, unknown>)[key] = value; + } + writeAtomic(path.join(root, FILE), Buffer.from(JSON.stringify(merged, null, 2) + "\n", "utf8")); + return { ...merged }; +} + +function writeAtomic(dest: string, bytes: Buffer): void { + const dir = path.dirname(dest); + const tmp = path.join(dir, `.${path.basename(dest)}.${process.pid}.tmp`); + fs.writeFileSync(tmp, bytes); + fs.renameSync(tmp, dest); +} diff --git a/extensions/office/src/store.ts b/extensions/office/src/store.ts new file mode 100644 index 000000000..5d06487f8 --- /dev/null +++ b/extensions/office/src/store.ts @@ -0,0 +1,730 @@ +/** + * The office document library: docs/<docId>/<title>.<ext> under the extension's own dataDir plus an + * index.json ({ version: 1, docs: DocMeta[] }) that is the single source of truth for metadata. Every + * operation reads the index fresh from disk (no caching -- a human UI tab, the loopback listener and + * the agent tools all share this process's files) and writes it back atomically (tmp file in the same + * directory + rename). Document bytes live in the plain filesystem: the portal's 1 MB managed-file cap + * does not apply here, so >1 MB Office files work natively. + * + * Created docs are written INTO this library (docs/<docId>/<title>.<ext>); workspace-scanned docs (see + * scanWorkspace) point at their file IN PLACE via DocMeta.path -- the scanned file IS the document, and + * every path-taking operation (persistBytes, renameDoc, deleteDoc, exportDoc, pruneMissingDocs) resolves + * through docPath(), which prefers that field. A deleted scanned doc moves its REAL file into this + * library's .trash (7-day sweep); a scanned file already gone out-of-band is pruned, not an error. + * + * Every exported function takes a trailing optional `root = dataDir()` so unit tests can pass a tmp + * dir (per the wave-2 amendments); production callers just omit it. The ONLY async functions are + * templateDocx/templateXlsx and createDoc, because jszip -- the only zip writer allowed in + * this package -- generates buffers asynchronously; everything that touches existing files is sync. + */ +import fs from "node:fs"; +import path from "node:path"; +import { randomInt } from "node:crypto"; +import { jsZip } from "./jszip"; // LAZY by design: the built server must import without its own node_modules yet (deps.js installs them at boot) +import { dataDir } from "./host"; +import { parseDocx } from "./ooxml"; // importFile + scanWorkspace baseline the source file's comment ids at import/index time (ping.ts first-save rule) + +export type DocMeta = { + docId: string; + title: string; + ext: "docx" | "xlsx"; + createdAt: number; + updatedAt: number; + sizeBytes: number; + mtimeMs: number; + owner: string | null; + grants: { chatId: string; commentId: string; at: number }[]; + createdBy?: string; + /** Set for workspace-scanned docs only (importFile stamps it too, for its copy flow). ping.ts uses it as the first-save rule: a scanned doc's pre-existing comments are baselined without wakes; a created (blank) doc gets no such baseline. */ + imported?: boolean; + /** Comment ids present in the file at INDEX time (stamped by scanWorkspace). The first-save baseline for scanned docs: any id NOT listed here appeared after indexing and is still processed normally. Absent when the source did not parse as docx. */ + importedBaseline?: string[]; + /** A workspace-scanned doc lives in PLACE: its real, absolute file path (realpathSync'd at index time). ABSENT for library-created docs, whose bytes are docs/<docId>/<title>.<ext> under the library root -- docPath() prefers this field when present. */ + path?: string; + /** Which workspace a scanned doc lives in -- the "workspace" sort key of the UI: the FIRST path segment under the scan + * root that contains it (scanWorkspace; deepest matching root wins, so /root/repos/pithagoras/... labels "pithagoras"), + * and for a file straight in a root that root's OWN dir name. Stamped at index time and BACKFILLED on every rescan; + * ABSENT for library-created docs -- the UI reads undefined as "library". */ + workspace?: string; +}; + +type IndexFile = { version: 1; docs: DocMeta[] }; + +const TRASH_TTL_MS = 7 * 24 * 60 * 60 * 1000; // sweepTrash removes .trash entries older than this +// importFile caps: the sync copy must not commit an unbounded file into the library (and any xlsx/docx in it +// is parsed later against ooxml.ts' 256 MiB / 1 GiB zip envelope). Above MAX_BASELINE_BYTES the docx comment +// baseline is skipped (ping.ts "unparseable source" fallback) instead of reading a huge file inline. +const MAX_IMPORT_BYTES = 512 * 1024 * 1024; +const MAX_BASELINE_BYTES = 32 * 1024 * 1024; + +/* ------------------------------------------------------------------ */ +/* index I/O */ +/* ------------------------------------------------------------------ */ + +function indexPath(root: string): string { + return path.join(root, "index.json"); +} + +function readIndex(root: string): IndexFile { + try { + const raw: unknown = JSON.parse(fs.readFileSync(indexPath(root), "utf8")); + if (typeof raw !== "object" || raw === null || !Array.isArray((raw as Record<string, unknown>).docs)) throw new Error("bad shape"); + return raw as IndexFile; + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") return { version: 1, docs: [] }; + throw Object.assign(new Error(`corrupt document index at ${indexPath(root)} -- run settleKept to reconcile`), { code: "index_corrupt", cause: err }); + } +} + +function writeIndex(root: string, index: IndexFile): void { + const dest = indexPath(root); + const tmp = path.join(root, `.${path.basename(dest)}.${process.pid}.tmp`); + fs.writeFileSync(tmp, JSON.stringify(index, null, 2)); + fs.renameSync(tmp, dest); +} + +function docPath(doc: DocMeta, root: string): string { + // A scanned doc's bytes are the workspace file itself; only library-created docs live under docs/<docId>/. + return doc.path ?? path.join(root, "docs", doc.docId, `${doc.title}.${doc.ext}`); +} + +function notFound(message = "no such document"): Error { + return Object.assign(new Error(message), { code: "not_found" }); +} + +/* ------------------------------------------------------------------ */ +/* identity + templates */ +/* ------------------------------------------------------------------ */ + +/** Title -> slug: lowercase, every run of non-alphanumeric characters collapses to one "-", leading/trailing "-" trimmed. + * A slug is not unique by itself (titles collide) -- createDoc appends a random suffix below. */ +export function slugify(title: string): string { + return title.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, ""); +} + +/** Six-character random suffix from crypto.randomInt over the lowercase ASCII alphanumerics "abcdefghijklmnopqrstuvwxyz0123456789" -- + * 36 symbols, so the chance that two freshly minted suffixes collide is ~1/36^6 (≈ 4.6e-10); createDoc still + * re-checks generated ids against the index and regenerates on the off chance of a hit. */ +function randomSuffix(): string { + const alphabet = "abcdefghijklmnopqrstuvwxyz0123456789"; + let out = ""; + for (let i = 0; i < 6; i++) out += alphabet[randomInt(alphabet.length)]; + return out; +} + +const DOCX_CONTENT_TYPES = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">` + + `<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>` + + `<Default Extension="xml" ContentType="application/xml"/>` + + `<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>` + + `<Override PartName="/word/styles.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml"/>` + + `</Types>`; + +const DOCX_ROOT_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/>` + + `</Relationships>`; + +const W_NS = `xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"`; + +// word/_rels/document.xml.rels is beyond the contract's four-part list but standard in every real docx and keeps +// Word/OnlyOffice happy (the styles part is referenced from it, not dangling) -- noted as an addition in the report. +const DOCX_DOC_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles" Target="styles.xml"/>` + + `</Relationships>`; + +const DOCX_DOCUMENT = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<w:document ${W_NS}><w:body><w:p><w:r><w:t xml:space="preserve">New document</w:t></w:r></w:p></w:body></w:document>`; + +const DOCX_STYLES = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><w:styles ${W_NS}></w:styles>`; + +/** A minimal valid docx (jszip, DEFLATE): [Content_Types].xml, _rels/.rels, word/document.xml (one paragraph + * saying "New document"), word/styles.xml -- plus word/_rels/document.xml.rels so the styles part is properly + * referenced. Opens cleanly in Word/OnlyOffice and parses with ooxml.parseDocx. jszip only writes buffers + * asynchronously, which is why this (and createDoc) are async. */ +export async function templateDocx(): Promise<Buffer> { + const zip = new (await jsZip())(); + zip.file("[Content_Types].xml", DOCX_CONTENT_TYPES); + zip.file("_rels/.rels", DOCX_ROOT_RELS); + zip.file("word/document.xml", DOCX_DOCUMENT); + zip.file("word/_rels/document.xml.rels", DOCX_DOC_RELS); + zip.file("word/styles.xml", DOCX_STYLES); + return zip.generateAsync({ type: "nodebuffer", compression: "DEFLATE" }); +} + +const XLSX_CONTENT_TYPES = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">` + + `<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>` + + `<Default Extension="xml" ContentType="application/xml"/>` + + `<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>` + + `<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>` + + `</Types>`; + +const XLSX_ROOT_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="xl/workbook.xml"/>` + + `</Relationships>`; + +const XLSX_WORKBOOK = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">` + + `<sheets><sheet name="Sheet1" sheetId="1" r:id="rId1"/></sheets>` + + `</workbook>`; + +const XLSX_WORKBOOK_RELS = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">` + + `<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet1.xml"/>` + + `</Relationships>`; + +const XLSX_SHEET1 = + `<?xml version="1.0" encoding="UTF-8" standalone="yes"?>` + + `<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData/></worksheet>`; + +/** A minimal valid xlsx (jszip, DEFLATE): [Content_Types].xml, _rels/.rels, xl/workbook.xml, xl/_rels/workbook.xml.rels + * and an empty xl/worksheets/sheet1.xml named "Sheet1". Opens cleanly in Excel/OnlyOffice and parses with + * ooxml.parseXlsx (one sheet, zero rows). Async for the same jszip reason as templateDocx. */ +export async function templateXlsx(): Promise<Buffer> { + const zip = new (await jsZip())(); + zip.file("[Content_Types].xml", XLSX_CONTENT_TYPES); + zip.file("_rels/.rels", XLSX_ROOT_RELS); + zip.file("xl/workbook.xml", XLSX_WORKBOOK); + zip.file("xl/_rels/workbook.xml.rels", XLSX_WORKBOOK_RELS); + zip.file("xl/worksheets/sheet1.xml", XLSX_SHEET1); + return zip.generateAsync({ type: "nodebuffer", compression: "DEFLATE" }); +} + +/* ------------------------------------------------------------------ */ +/* library operations */ +/* ------------------------------------------------------------------ */ + +/** All documents, titles ascending (localeCompare). */ +export function listDocs(root = dataDir()): DocMeta[] { + return [...readIndex(root).docs].sort((a, b) => a.title.localeCompare(b.title)); +} + +/** One document by id, or null. */ +export function getDoc(docId: string, root?: string): DocMeta | null { + const rootDir = root ?? dataDir(); + return readIndex(rootDir).docs.find((d) => d.docId === docId) ?? null; +} + +/** The ONE index writer for mention grants (ping.ts used to read-modify-write index.json on its own): fresh + * read, dedupe by (chatId, commentId) -- a re-sent comment never adds a second row -- atomic tmp+rename through + * the store's writeIndex. Sync like everything here and no logging (module contract). Returns "added", + * "duplicate" (already granted: nothing written), or null when the doc is not in the index any more. + * A corrupt index throws the usual {code:"index_corrupt"} -- a bad file, not a vanished doc. */ +export function addGrant(docId: string, chatId: string, commentId: string, at: number, root = dataDir()): "added" | "duplicate" | null { + const index = readIndex(root); + const doc = index.docs.find((d) => d.docId === docId); + if (!doc) return null; + const grants = (doc.grants ??= []); + if (grants.some((g) => g.chatId === chatId && g.commentId === commentId)) return "duplicate"; + grants.push({ chatId, commentId, at }); + writeIndex(root, index); + return "added"; +} + +/** The documents a home chat sees in its panel and office tools (plan 2.1, extended): every UNOWNED doc -- the + * workspace-scanned and human-made docs (owner null), shared by definition and visible to all homes -- plus ones it + * OWNS or holds a mention-GRANT for; deduped by docId (a doc owned and granted to the same home appears once). */ +export function docsForHome(homeChatId: string, root = dataDir()): DocMeta[] { + const seen = new Set<string>(); + const out: DocMeta[] = []; + for (const doc of readIndex(root).docs) { + if (seen.has(doc.docId)) continue; + if (doc.owner === null || doc.owner === homeChatId || doc.grants.some((g) => g.chatId === homeChatId)) { + seen.add(doc.docId); + out.push(doc); + } + } + return out.sort((a, b) => a.title.localeCompare(b.title)); +} + +function freshDocId(existing: DocMeta[], title: string): string { + const slug = slugify(title); + let docId = `${slug ? slug + "-" : ""}${randomSuffix()}`; // titles of pure punctuation still get a clean id + while (existing.some((d) => d.docId === docId)) docId = `${slug ? slug + "-" : ""}${randomSuffix()}`; + return docId; +} + +/** Titles become FILE NAMES (docs/<docId>/<title>.<ext>): reject anything empty, over 200 chars, or carrying a + * path separator / C0 control character -- "../../etc/passwd.docx" would otherwise escape the library. */ +function assertSafeTitle(title: string): void { + if (title === "" || title.length > 200) { + throw Object.assign(new Error(`a document title must be 1-200 characters: ${JSON.stringify(title.slice(0, 80))}`), { code: "invalid_title" }); + } + for (const ch of title) { + const code = ch.codePointAt(0) as number; + if (ch === "/" || ch === "\\" || code < 0x20) throw Object.assign(new Error(`the document title may not contain path separators or control characters: ${JSON.stringify(title.slice(0, 80))}`), { code: "invalid_title" }); + } +} + +function storeDoc(doc: DocMeta, bytes: Buffer, rootDir: string): DocMeta { // sync on purpose: the create/import paths must not await between readIndex and writeIndex + const dir = path.join(rootDir, "docs", doc.docId); + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, `${doc.title}.${doc.ext}`); + fs.writeFileSync(file, bytes); + const st = fs.statSync(file); + doc.sizeBytes = st.size; + doc.mtimeMs = st.mtimeMs; + return doc; +} + +/** Create a document from the blank template of its type. docId = slug(title) + "-" + 6 random chars (see + * randomSuffix for the alphabet). Duplicate -- an existing entry with the SAME docId or the SAME title AND same + * type (case-sensitive) -- throws Error{code:"exists"} before anything touches disk. The owner is the creating + * home chat (agent tools pass it) or null for human-created docs; createdBy records who actually did it. */ +export async function createDoc(input: { title: string; type: "word" | "cell"; owner?: string | null; createdBy?: string }, root = dataDir()): Promise<DocMeta> { + const rootDir = root ?? dataDir(); + assertSafeTitle(input.title); // titles are file names + const ext: DocMeta["ext"] = input.type === "word" ? "docx" : "xlsx"; + const bytes = ext === "docx" ? await templateDocx() : await templateXlsx(); // BEFORE the index round-trip: no await may sit between readIndex and writeIndex, or a concurrent create would have its write clobbered + const index = readIndex(rootDir); + if (index.docs.some((d) => d.title === input.title && d.ext === ext)) throw Object.assign(new Error(`a document titled "${input.title}" already exists`), { code: "exists" }); + const now = Date.now(); + const doc: DocMeta = { + docId: freshDocId(index.docs, input.title), + title: input.title, + ext, + createdAt: now, + updatedAt: now, + sizeBytes: 0, + mtimeMs: 0, + owner: input.owner ?? null, + grants: [], + }; + if (input.createdBy !== undefined) doc.createdBy = input.createdBy; + storeDoc(doc, bytes, rootDir); + index.docs.push(doc); + writeIndex(rootDir, index); + return { ...doc }; +} + +/** Retitle a document and rename its file on disk (docs/<docId>/<newTitle>.<ext>). The docId is stable across the + * whole lifetime of the doc (tickets, history paths, state files key off it), and rename preserves the content's + * mtimeMs -- that IS the revision tools gate on, so a retitle is never a new revision. updatedAt bumps. A title + * already held by ANOTHER document throws {code:"exists"} (the plan does not specify retitle collisions; an + * exists refusal is the safe reading). */ +export function renameDoc(docId: string, newTitle: string, root = dataDir()): DocMeta { + const rootDir = root ?? dataDir(); + assertSafeTitle(newTitle); + const index = readIndex(rootDir); + const doc = index.docs.find((d) => d.docId === docId); + if (!doc) throw notFound(`no document with id ${docId}`); + if (newTitle !== doc.title && index.docs.some((d) => d.docId !== docId && d.title === newTitle)) { + throw Object.assign(new Error(`a document titled "${newTitle}" already exists`), { code: "exists" }); + } + // One rule for both kinds: rename inside the file's OWN directory. For a library doc that is docs/<docId>/; + // for a scanned doc it renames the real workspace file next to where it already is (same parent, new name). + const from = docPath(doc, rootDir); + const to = path.join(path.dirname(from), `${newTitle}.${doc.ext}`); + fs.renameSync(from, to); + if (doc.path) doc.path = to; + doc.title = newTitle; + doc.updatedAt = Date.now(); + writeIndex(rootDir, index); + return { ...doc }; +} + +/** Delete a document: its docs/<docId> folder is moved to .trash/<Date.now()>-<docId> (a hard delete would not be + * undoable) and the index entry removed. A scanned doc's REAL FILE (not a library folder) is what moves -- deleting + * one moves the workspace file into this library's trash, exactly as reversible as everything else here. A scanned + * file already gone from its workspace (deleted out-of-band since the last scan) is NOT an error: only the index + * entry drops, which is what pruneMissingDocs would have done at the next settleKept anyway. sweepTrash (server boot / + * settleKept) reclaims trash older than 7 days. Throws {code:"not_found"} for an unknown id. */ +export function deleteDoc(docId: string, root = dataDir()): void { + const rootDir = root ?? dataDir(); + const index = readIndex(rootDir); + const doc = index.docs.find((d) => d.docId === docId); + if (!doc) throw notFound(`no document with id ${docId}`); + const from = doc.path ? doc.path : path.join(rootDir, "docs", doc.docId); + fs.mkdirSync(trashDir(rootDir), { recursive: true }); + try { + fs.renameSync(from, path.join(trashDir(rootDir), `${Date.now()}-${docId}`)); // the real file (scanned) or the folder (library) + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== "ENOENT") throw err; + // the file left its workspace on its own: nothing to move, the index drop below IS the deletion + } + writeIndex(rootDir, { version: 1, docs: index.docs.filter((d) => d.docId !== docId) }); +} + +/** The library's trash dir, made if missing -- one place that names it. */ +function trashDir(rootDir: string): string { + return path.join(rootDir, ".trash"); +} + +/** Reclaim .trash entries whose name timestamp (the "<ms>-<name>" prefix written by deleteDoc) is older than 7 days. + * Returns how many were removed. Server boot (settleKept) and the lookAfter path call this. */ +export function sweepTrash(root = dataDir()): number { + const rootDir = root ?? dataDir(); + const trash = trashDir(rootDir); + let entries: string[]; + try { + entries = fs.readdirSync(trash); + } catch { + return 0; // no .trash yet + } + const now = Date.now(); + let removed = 0; + for (const name of entries) { + const dash = name.indexOf("-"); + if (dash < 1) continue; // not a deleteDoc entry -- leave it alone + const stamp = Number(name.slice(0, dash)); + if (!Number.isFinite(stamp)) continue; + if (now - stamp > TRASH_TTL_MS) { + fs.rmSync(path.join(trash, name), { recursive: true, force: true }); + removed++; + } + } + return removed; +} + +/** The settleKept half of index-vs-disk reconciliation (server.ts calls it at boot and on the interval): drop the + * index entries whose library file is gone from disk -- someone deleted a docs/<docId>/ dir out-of-band, or its bytes + * were lost. The index is the source of truth in BOTH directions: a docs/<id>/ dir with NO index entry is NOT added + * (an orphan sweep is not v1). Returns the dropped docIds; writes nothing when nothing is missing. */ +export function pruneMissingDocs(root = dataDir()): string[] { + const rootDir = root ?? dataDir(); + const index = readIndex(rootDir); + const kept: DocMeta[] = []; + const dropped: string[] = []; + for (const doc of index.docs) { + try { + fs.accessSync(docPath(doc, rootDir)); + kept.push(doc); + } catch { + dropped.push(doc.docId); + } + } + if (dropped.length) writeIndex(rootDir, { version: 1, docs: kept }); + return dropped; +} + +/** ONE file of a scan, resolved to its index entry: throws {code:"unsupported_format"} for anything that is not a + * regular .docx/.xlsx and {code:"too_large"} above the 512 MiB cap; returns null when the basename (without its + * extension) cannot become a document title -- such a file is skipped, not indexed. The single place where a + * scanned file becomes a DocMeta, so the stamping rules of the former importFile live in exactly one spot. */ +function indexOne(real: string, rootDir: string, workspace?: string): DocMeta | null { + const base = path.basename(real); + const dot = base.lastIndexOf("."); + if (dot < 1) throw Object.assign(new Error(`${base} has no extension`), { code: "unsupported_format" }); + const extRaw = base.slice(dot + 1).toLowerCase(); + if (extRaw !== "docx" && extRaw !== "xlsx") throw Object.assign(new Error(`${base}: only .docx and .xlsx files are indexed`), { code: "unsupported_format" }); + let st: fs.Stats; + try { + st = fs.statSync(real); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") throw notFound(`no such file ${real}`); + throw err; + } + if (!st.isFile()) throw Object.assign(new Error(`${real} is not a regular file`), { code: "unsupported_format" }); + // The cap importFile had, now on INDEXING: an uncapped source would also blow the ooxml reader's 1 GiB zip + // envelope at parse time -- above it the file stays out of the library and its mtime-gated flow. + if (st.size > MAX_IMPORT_BYTES) throw Object.assign(new Error(`${real} is ${st.size} bytes: files above ${MAX_IMPORT_BYTES / (1024 * 1024)} MiB are not indexed`), { code: "too_large" }); + const title = base.slice(0, dot); + try { + assertSafeTitle(title); // a file name that cannot be a document title is skipped, not indexed + } catch { + return null; + } + // Baseline AT INDEX TIME (the importFile rule, carried over): record which comment ids this file carries NOW, so + // pre-existing @mentions do not fire wakes at index time and the first save baselines exactly this history. Only + // for files up to MAX_BASELINE_BYTES -- above it the field stays undefined and ping.ts applies its documented + // "unparseable source" fallback; a docx that does not parse leaves it undefined too. + let importedBaseline: string[] | undefined; + if (extRaw === "docx" && st.size <= MAX_BASELINE_BYTES) { + try { + importedBaseline = parseDocx(fs.readFileSync(real)).comments.map((c) => c.id); + } catch { + // does not parse as docx: baseline unknown, ping.ts' fallback applies + } + } + const now = Date.now(); + const index = readIndex(rootDir); + if (index.docs.some((d) => d.path === real || (d.title === title && d.ext === extRaw))) { + throw Object.assign(new Error(`a document titled "${title}" (${extRaw}) is already in the library`), { code: "exists" }); + } + const doc: DocMeta = { + docId: `${slugify(title)}-${randomSuffix()}`, + title, + ext: extRaw, + createdAt: now, + updatedAt: now, + sizeBytes: st.size, + mtimeMs: st.mtimeMs, // the file's OWN mtime IS its revision from day one -- the mtime-gated flow edits it in place + owner: null, // a human doc: no home owns it -- it shows in the global library, exactly like a human-made created doc + grants: [], + imported: true, // ping.ts first-save rule for the stamped baseline above + path: real, // IN PLACE: the scanned file IS the document; nothing is copied into the library + }; + if (importedBaseline !== undefined) doc.importedBaseline = importedBaseline; + if (workspace !== undefined) doc.workspace = workspace; // the "workspace" sort label of the UI (scanWorkspace) + index.docs.push(doc); + writeIndex(rootDir, index); + return { ...doc }; +} + +/** Synchronous workspace scan: every *.docx/*.xlsx (case-insensitive) under `roots`, indexed IN PLACE. + * + * roots are the portal's agent folders (portal().folders.list(): every agent home, every project, every chat's + * working folder). The walk is bounded and deliberately dumb -- no git, no content sniffing: it never enters the + * library's own data dir (indexing a copy of itself would re-index its own docs forever), never hidden dirs + * (dot-prefixed: .git, editor scratch), and never node_modules. A file is identified by its REALPATH, so two roots + * reaching the same file through a symlink or bind mount produce ONE entry. Already-indexed files get their + * mtimeMs/sizeBytes REFRESHED from disk (a rescan after something wrote the file straight to disk keeps the list + * honest) and count as `updated` only when something actually changed; new files go through indexOne -- in place, + * owner null, with the importedBaseline stamp so pre-existing @mentions do not fire wakes at index time. A file + * that cannot be indexed (oversized, unsafe title, already titled, unreadable) is skipped: one bad file must + * never fail the whole sweep. + * + * Returns { added, updated }. */ +export function scanWorkspace(roots: string[], root = dataDir()): { added: number; updated: number } { + const rootDir = root ?? dataDir(); + let self: string; + try { + self = fs.realpathSync(rootDir); + } catch { + self = path.resolve(rootDir); + } + const seenDirs = new Set<string>(); // per-walk guard: a dir is walked at most once (also breaks symlink loops) + const found: string[] = []; // candidate files, in walk order + + const isSkippedName = (name: string): boolean => name.startsWith(".") || name === "node_modules"; + const walk = (dir: string): void => { + let real: string; + try { + real = fs.realpathSync(dir); + } catch { + return; // gone mid-walk or unreadable -- the rest of the tree still counts + } + if (real === self || seenDirs.has(real)) return; // our own library dir is never walked, and neither is any dir twice + seenDirs.add(real); + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const entry of entries) { + const at = path.join(dir, entry.name); + if (entry.isDirectory()) { + if (!isSkippedName(entry.name)) walk(at); // hidden dirs and node_modules are never walked + continue; + } + if (entry.isFile() && /\.(docx|xlsx)$/i.test(entry.name)) found.push(at); + } + }; + for (const r of roots) walk(r); + + // A file's WORKSPACE label (the "workspace" sort key of the UI): the first path segment under whichever scan root + // contains it -- the DEEPEST matching root wins, so a nested root labels more specifically than its parent. + // A file straight in a root takes that root's OWN dir name instead of being labelled by its own file name + // (a home-root file -> the home's dir name). undefined when no resolved root claims the file. + const resolvedRoots = roots.map((r) => { + try { + return fs.realpathSync(r); + } catch { + return path.resolve(r); + } + }); + const workspaceOf = (real: string): string | undefined => { + let best: { dir: string; rel: string } | null = null; + for (const rootReal of resolvedRoots) { + if (real === rootReal || !real.startsWith(rootReal + path.sep)) continue; + const rel = path.relative(rootReal, real); + if (!best || rootReal.length > best.dir.length) best = { dir: rootReal, rel }; + } + if (!best) return undefined; + const segs = best.rel.split(path.sep); + return segs.length > 1 ? segs[0] : path.basename(best.dir); + }; + + // What is ALREADY indexed, by its real path: refreshes target these; everything else in `found` is a candidate add. + const byReal = new Map<string, DocMeta>(); + const initial = readIndex(rootDir); + for (const doc of initial.docs) if (doc.path) byReal.set(doc.path, doc); + let added = 0; + let updated = 0; + for (const file of found) { + let real: string; + try { + real = fs.realpathSync(file); // symlinks collapse into one identity; a broken link is skipped + } catch { + continue; + } + const known = byReal.get(real); + if (known) { + let st: fs.Stats; + try { + st = fs.statSync(real); + } catch { + continue; // vanished since the index was read: pruneMissingDocs settles it at the next settleKept + } + if (st.mtimeMs !== known.mtimeMs || st.size !== known.sizeBytes) { + known.mtimeMs = st.mtimeMs; + known.sizeBytes = st.size; + known.updatedAt = Date.now(); + updated++; + } + const ws = workspaceOf(real); + if (ws !== undefined && known.workspace !== ws) { + // BACKFILL on every rescan: entries indexed before the stamp (or under an older root layout) pick up their + // workspace label here -- counted as a refresh, exactly once. + known.workspace = ws; + known.updatedAt = Date.now(); + updated++; + } + continue; // refresh-and-continue: a rescan never re-indexes what it already has + } + try { + const doc = indexOne(real, rootDir, workspaceOf(real)); // its own atomic read-modify-write -- one entry per file + if (doc) { + byReal.set(real, doc); + added++; + } + } catch (err) { + const ce = err as { code?: string }; + if (ce.code === "unsupported_format" || ce.code === "too_large" || ce.code === "exists") continue; // expected skips + throw err; // a real failure (index corrupt, disk full) IS a rescan failure + } + } + if (updated > 0) { + const index = readIndex(rootDir); // fresh: add-writes may have landed since `initial` + let wrote = false; + for (const [realPath, doc] of byReal) { + const entry = index.docs.find((d) => d.path === realPath); + if ( + !entry || + (doc.mtimeMs === entry.mtimeMs && doc.sizeBytes === entry.sizeBytes && doc.workspace === entry.workspace) + ) continue; + entry.mtimeMs = doc.mtimeMs; + entry.sizeBytes = doc.sizeBytes; + entry.updatedAt = doc.updatedAt; + if (doc.workspace !== undefined) entry.workspace = doc.workspace; + wrote = true; + } + if (wrote) writeIndex(rootDir, index); + } + return { added, updated }; +} + +/** Import an existing .docx/.xlsx from an absolute path into the library (copy, never move -- the source stays). SUPERSEDED for the UI flow by scanWorkspace (in-place indexing); retained because ping.test exercises its copy-into-library shape. + * The extension comes from the basename (case-insensitive) and anything else throws {code:"unsupported_format"}; + * a missing/unreadable source throws {code:"not_found"}; a file above 512 MiB throws {code:"too_large"} before + * anything is copied. title defaults to the basename without its extension; owner is set by the caller (an + * agent/project context passes its home chat, human imports stay ownerless). For docx sources up to 32 MiB the + * comment ids are baselined into importedBaseline at import time; above that the field stays undefined and + * ping.ts applies its "unparseable source" first-save fallback. */ +export function importFile(absPath: string, meta: { title?: string; owner?: string | null }, root = dataDir()): DocMeta { + const rootDir = root ?? dataDir(); + const base = path.basename(absPath); + const dot = base.lastIndexOf("."); + if (dot < 1) throw Object.assign(new Error(`${base} has no extension`), { code: "unsupported_format" }); + const extRaw = base.slice(dot + 1).toLowerCase(); + if (extRaw !== "docx" && extRaw !== "xlsx") throw Object.assign(new Error(`${base}: only .docx and .xlsx files can be imported`), { code: "unsupported_format" }); + let st: fs.Stats; + try { + st = fs.statSync(absPath); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") throw notFound(`no such file ${absPath}`); + throw err; + } + if (!st.isFile()) throw Object.assign(new Error(`${absPath} is not a regular file`), { code: "unsupported_format" }); + // Hard size cap (N2): importFile copies synchronously into the library and parses xlsx/docx later -- an uncapped + // source would also blow the ooxml reader's 1 GiB zip envelope. 512 MiB is far above any real Office file. + if (st.size > MAX_IMPORT_BYTES) throw Object.assign(new Error(`${absPath} is ${st.size} bytes: imports are capped at ${MAX_IMPORT_BYTES / (1024 * 1024)} MiB`), { code: "too_large" }); + const title = meta.title ?? base.slice(0, dot); + assertSafeTitle(title); // the imported file lands in the library as <title>.<ext> -- same rules as created titles + // Baseline AT IMPORT TIME (review 4364810146): record which comment ids this file carries NOW, so a mention added to it + // before its first save is still NEW at first save and can wake; only the history that came in WITH the file baselines. + // scanWorkspace stamps the same baseline IN PLACE via indexOne -- keep the two stampings in step. + // Only for files up to MAX_BASELINE_BYTES: reading a multi-hundred-MiB docx inline just to stamp a baseline is not worth + // the sync stall -- above the cap leave it undefined so ping.ts applies its documented "unparseable source" fallback + // (baseline everything the first save holds) and first-save wake semantics are preserved. + let importedBaseline: string[] | undefined; + if (extRaw === "docx" && st.size <= MAX_BASELINE_BYTES) { + try { + importedBaseline = parseDocx(fs.readFileSync(absPath)).comments.map((c) => c.id); + } catch { + // source does not parse as docx: leave the baseline unknown -- ping.ts then falls back to baselining whatever the first save holds + } + } + const index = readIndex(rootDir); + const now = Date.now(); + const doc: DocMeta = { + docId: `${slugify(title)}-${randomSuffix()}`, // fresh id -- imports are copies; the index's exists rule (title+type) is NOT re-checked for imports, so a same-titled import coexists + title, + ext: extRaw, + createdAt: now, + updatedAt: now, + sizeBytes: st.size, + mtimeMs: 0, + owner: meta.owner ?? null, + grants: [], + imported: true, // ping.ts first-save rule: the import-time baseline below is what gets baselined, never woken + importedBaseline, + }; + const dir = path.join(rootDir, "docs", doc.docId); + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, `${title}.${extRaw}`); + fs.copyFileSync(absPath, file); + const dst = fs.statSync(file); + doc.sizeBytes = dst.size; + doc.mtimeMs = dst.mtimeMs; + index.docs.push(doc); + writeIndex(rootDir, index); + return { ...doc }; +} + +/** Copy a document OUT of the library to destAbsDir (created if missing) as <title>.<ext>; returns the absolute + * path written. Throws {code:"not_found"} for an unknown id. */ +export function exportDoc(docId: string, destAbsDir: string, root = dataDir()): string { + const rootDir = root ?? dataDir(); + const doc = getDoc(docId, rootDir); + if (!doc) throw notFound(`no document with id ${docId}`); + fs.mkdirSync(destAbsDir, { recursive: true }); + const dest = path.join(destAbsDir, `${doc.title}.${doc.ext}`); + fs.copyFileSync(docPath(doc, rootDir), dest); + return path.resolve(dest); +} + +/** Persist saved document bytes (the callback's status-2/6 path) under the REVISION GATE -- the whole point of + * mtime-as-revision: stat the library file FIRST and if its mtimeMs differs from doc.mtimeMs (tolerance none; + * the caller passes the meta it last saw), refuse with Error{code:"revision_mismatch", currentMtimeMs} so a stale + * DS session can never clobber an agent edit (or vice versa). On success: temp file in the same directory + rename + * over the original (the existing bytes are never deleted mid-write), re-stat for the new mtimeMs/sizeBytes, update + * the index entry (mtimeMs, sizeBytes, updatedAt) and return { mtimeMs } -- the caller answers DS and registers the + * history revision under that number. Sync on purpose: no zip is involved here, only a byte copy. */ +export function persistBytes(doc: DocMeta, bytes: Buffer, root = dataDir()): { mtimeMs: number } { + const rootDir = root ?? dataDir(); + const file = docPath(doc, rootDir); + let st: fs.Stats; + try { + st = fs.statSync(file); + } catch { + throw notFound(`no library file for ${doc.docId}`); + } + if (st.mtimeMs !== doc.mtimeMs) { + throw Object.assign(new Error("revision_mismatch: the document changed since this session opened it"), { code: "revision_mismatch", currentMtimeMs: st.mtimeMs }); + } + const tmp = path.join(path.dirname(file), `.${path.basename(file)}.${process.pid}.tmp`); + fs.writeFileSync(tmp, bytes); + fs.renameSync(tmp, file); + const after = fs.statSync(file); + const index = readIndex(rootDir); + const entry = index.docs.find((d) => d.docId === doc.docId); + if (!entry) throw notFound(`no document with id ${doc.docId}`); + entry.mtimeMs = after.mtimeMs; + entry.sizeBytes = after.size; + entry.updatedAt = Date.now(); + writeIndex(rootDir, index); + return { mtimeMs: after.mtimeMs }; +} diff --git a/extensions/office/src/tickets.ts b/extensions/office/src/tickets.ts new file mode 100644 index 000000000..e462753b0 --- /dev/null +++ b/extensions/office/src/tickets.ts @@ -0,0 +1,56 @@ +/** + * Document-fetch tickets for the DS container: document.url in a minted editor config points at the + * loopback listener's GET /doc/<ticket>, which carries no session cookie (the portal gates every /api + * route, and the DS holds none). The token is self-authenticating -- base64url("docId|expMs|hmac") with + * a 10-minute default expiry -- verified here in constant time. Pure: no fs, no portal; the caller owns + * the secret string (the same jwt-secret value as jwt.ts, per plan 5.6). Purpose separation (S3): the ticket + * has no JWT claims to carry a family marker in, so the marker lives INSIDE the MAC'd message -- the HMAC + * is computed over "document-ticket|docId|expMs" and only a verifier of this same family knows to prefix it. + */ + +/** The purpose domain separator inside every ticket MAC. Mint and verify both prepend it; changing it here + * invalidates outstanding tickets (10-minute lifetime at most) -- deliberately, like rotating the secret. */ +const TICKET_PURPOSE = "document-ticket"; +import { createHmac, timingSafeEqual } from "node:crypto"; + +function b64url(bytes: Buffer | string): string { + return Buffer.from(typeof bytes === "string" ? bytes : bytes).toString("base64url"); +} + +function hmacHex(secret: string, message: string): string { + return createHmac("sha256", secret).update(message).digest("hex"); +} + +/** Mint a fetch ticket for one document. token = base64urlNoPad(`docId|<expiresMs>|<hmacSha256Hex(secret, TICKET_PURPOSE + "|" + docId + "|" + expiresMs)>)`; + * expiresMs is Date.now() + ttlMs (default 10 minutes). */ +export function mintTicket(docId: string, ttlMs = 600_000, secret: string): { token: string; expiresMs: number } { + if (secret === "") throw new Error("mintTicket: the shared secret is required"); // an empty secret would let anyone forge a ticket + const expiresMs = Date.now() + ttlMs; + return { token: b64url(`${docId}|${expiresMs}|${hmacHex(secret, `${TICKET_PURPOSE}|${docId}|${expiresMs}`)}`), expiresMs }; +} + +/** Verify a ticket: split on "|", recompute the HMAC over TICKET_PURPOSE|docId|exp and compare the hex digests in + * constant time (timingSafeEqual over Buffer.from(hex, "hex")), then require the expiry to be in the future. + * Returns { docId } or null -- never throws. A token whose payload splits into anything other than three + * non-empty sections is rejected outright (so a docId containing "|" can never verify either). A MAC computed + * WITHOUT the purpose prefix (or for another family) fails the digest comparison. */ +export function verifyTicket(token: string, secret: string): { docId: string } | null { + if (secret === "") return null; // an empty secret would make every well-formed token verify -- refuse instead of forging + if (!/^[A-Za-z0-9_-]*$/.test(token)) return null; // base64url alphabet only + let decoded: Buffer; + try { + decoded = Buffer.from(token, "base64url"); + } catch { + return null; + } + const parts = decoded.toString("utf8").split("|"); + if (parts.length !== 3 || parts.some((section) => section === "")) return null; // exactly three non-empty sections + const [docId, expStr, mac] = parts; + const expiresMs = Number(expStr); + if (!Number.isFinite(expiresMs) || !/^[0-9a-f]+$/i.test(mac)) return null; + const expected = Buffer.from(hmacHex(secret, `${TICKET_PURPOSE}|${docId}|${expStr}`), "hex"); + const given = Buffer.from(mac.toLowerCase(), "hex"); + if (given.length !== expected.length || !timingSafeEqual(given, expected)) return null; + if (expiresMs <= Date.now()) return null; // expiry is not in the future + return { docId }; +} diff --git a/extensions/office/src/tools.ts b/extensions/office/src/tools.ts new file mode 100644 index 000000000..b54854d19 --- /dev/null +++ b/extensions/office/src/tools.ts @@ -0,0 +1,451 @@ +/** + * The agent's office tools (plan 5.5): list, create, read, edit and list the comments of the documents + * in this home's office-library scope (store.docsForHome -- every UNOWNED doc: the workspace-scanned and + * human-made docs are shared by all homes -- plus what this home owns or was granted), docs-extension-style + * -- named by id, base-revision writes, coded error prefixes ("error:<code> <message>", the exact + * docs convention with the "error:" prefix the office contract fixes). + * + * Ownership scope (plan 2.1): `registerTools` walks every chat through portal.chats.extend; for a + * chat it resolves the workspace folder (chats.folder(chat.id)), finds it in folders.list() by path + * and REGISTER ONLY when its kind is "agent" or "project". The tool scope is then docsForHome(home), + * where `home` is the folder's home chat: the sessions working in that folder (chats.inFolder) are + * its home chats, deterministically picked as the lowest id -- exactly ping.ts' resolveMention rule, + * so a doc created here is owned by the same chat a mention wakes it to. Docs outside the scope are + * invisible to every tool: office_read asks for them and hears the same "no document with that id" + * as for a nonexistent one (docs convention -- no other chat's documents leak through). + * + * Revision model (plan 5.5/5.7): the mtimeMs of the library file IS the revision. office_edit + * takes baseRevision (= the mtimeMs office_read showed), GATES FIRST (>1 ms drift -> + * "error:revision_mismatch ..." with the fresh mtime, nothing written), and store.persistBytes re- + * gates strictly at write time anyway (ZERO tolerance: stat-vs-meta must match exactly). The agent is + * a CO-EDITOR alongside human editor sessions (live.ts) -- like Office 365 there is NO switch and NO + * session check: the edit proceeds under that gate regardless of who else has the document open. If the + * file moved on in between (a co-editor's save), NOTHING is written and the tool answers a loud CONFLICT + * ("error:revision_mismatch ...", naming the open editor session when one stands) -- re-read and retry. + * A stale baseRevision always fails that way. + * + * Wakes (plan 5.4 step 4 -- "the drain side"): ping.ts only QUEUES wakes server-side and rings the + * bell with chats.tell(chatId, {type:"ping"}). The office pi extension registered from THIS factory + * listens on the UI_RECEIVE bus event ("pithagoras:v1:ui:receive", the core/docs constant) and drains + * the queue per entry via pi.sendMessage({customType:"office-comment-ping", ...}, + * {deliverAs:"followUp", triggerTurn:true}), docs' hearPings pattern. + * + * Injection seams (unit tests must never reach portal()): registerTools(opts) takes the portal bits + * -- chats.extend registrar, chat->folder lookup, folders list, folder->sessions, store root, audit + * record fn, log fn. EVERY one defaults LAZY: the real portal() is read only when a default is used + * (in production). Mutating ops (office_create, office_edit) call record({subject, reason, chatId}) + * after the write; the default seam is a one-line log via portal().log (ping.ts pattern -- T-WIRE may + * pass a real audit sink). + */ +import fs from "node:fs"; +import path from "node:path"; +import { jsZip } from "./jszip"; // LAZY by design: the built server must import without its own node_modules yet (deps.js installs them at boot) +import { portal } from "./host"; +import { getDoc, docsForHome, createDoc, persistBytes, type DocMeta } from "./store"; +import { parseDocx, parseXlsx, editDocx, rewriteZip, type EditOp } from "./ooxml"; +import { usersOf, isLocked } from "./live"; +import { drainWakeQueue } from "./ping"; + +/** The UI_RECEIVE bus event chats.tell rings (core server extension-ui.ts and docs/ping.js use the same name). */ +const UI_RECEIVE = "pithagoras:v1:ui:receive"; + +/** An audit line recorded after every mutating op. `subject` is the doc title, `chatId` the acting home chat. T-WIRE may route this to a real audit sink. */ +export type OfficeAuditEntry = { subject: string; reason: string; chatId: string }; + +/** The folders list entries the factory resolves against (the shape of portal.folders.list() minus nothing it needs). */ +export type ToolsFolderInfo = { path: string; name: string; kind: "agent" | "project" | "chat"; home?: boolean }; + +/** Injection seams for registerTools. Every field is optional and LAZY -- defaults read portal() only when actually used, so unit tests pass plain values and never reach the real portal. */ +export type ToolsDeps = { + /** The chats.extend registrar. Default: (factory) => portal().chats.extend(factory). */ + extend?: (factory: (pi: unknown, chat: { id: string }) => void) => void; + /** A chat's workspace folder path (throws when the chat or its folder is gone). Default: (id) => portal().chats.folder(id). */ + chatFolder?: (chatId: string) => string; + /** The folders the agents work in. Default: () => portal().folders.list(). */ + folders?: () => ToolsFolderInfo[]; + /** The sessions working in a folder path (home-chat resolution, plan 2.1). Default: (p) => portal().chats.inFolder(p). */ + inFolder?: (folderPath: string) => { id: string }[]; + /** Store/state root override. Default: lazy portal().dataDir(). */ + root?: string; + /** The audit sink for mutating ops. Default: a one-line log via the log seam (ping.ts pattern). */ + record?: (entry: OfficeAuditEntry) => void; + /** Log line. Default: lazy portal().log. */ + log?: (line: string) => void; +}; + +let cfg: ToolsDeps = {}; + +/** Test seam: drop the module-level seams (the tests' injections are back to the lazy portal defaults). */ +export function __resetToolsForTests(): void { + cfg = {}; +} + +/* ------------------------------------------------------------------ */ +/* lazy defaults (portal() only when actually used) */ +/* ------------------------------------------------------------------ */ + +function rootDir(): string { + return cfg.root ?? portal().dataDir(); +} +function chatFolder(chatId: string): string { + return (cfg.chatFolder ?? ((id: string) => portal().chats.folder(id)))(chatId); +} +function folders(): ToolsFolderInfo[] { + return (cfg.folders ?? (() => portal().folders.list()))(); +} +function inChats(folderPath: string): { id: string }[] { + return (cfg.inFolder ?? ((p: string) => portal().chats.inFolder(p)))(folderPath); +} +function logLine(line: string): void { + (cfg.log ?? ((l: string) => portal().log(l)))(line); +} +function record(entry: OfficeAuditEntry): void { + // Lazy default is a one-line log (T-WIRE may pass a real audit sink later via registerTools({record})). + (cfg.record ?? ((e: OfficeAuditEntry) => logLine(`office audit: ${e.subject}: ${e.reason} (chat ${e.chatId})`)))(entry); +} + +/* ------------------------------------------------------------------ */ +/* registration */ +/* ------------------------------------------------------------------ */ + +/** + * server.ts calls ONCE at boot. Later calls (unit tests) update the seams and register again through + * whatever registrar is configured -- production registers exactly once, tests each get their own fake + * `extend`, so no once-guard here (unlike wirePing's onPersisted hook, which truly can only attach once). + */ +export function registerTools(opts?: ToolsDeps): void { + if (opts) cfg = { ...cfg, ...opts }; + const registrar = cfg.extend ?? ((factory: (pi: unknown, chat: { id: string }) => void) => portal().chats.extend(factory)); + registrar((pi, chat) => installForChat(pi, chat)); +} + +/** The chats.extend factory body (contract): resolve the chat's folder via chats.folder -> folders.list() by path -> kind; RETURN UNLESS "agent" or "project"; then register the tools on that chat's pi. */ +function installForChat(pi: unknown, chat: { id: string }): void { + let folderPath: string; + try { + folderPath = chatFolder(chat.id); + } catch (e) { + logLine(`office tools: no workspace folder for chat ${chat.id} (${(e as Error)?.message ?? e}) -- no office_* tools`); + return; // gone chat or folder: nothing to register into + } + const folder = folders().find((f) => f.path === folderPath); + if (!folder || (folder.kind !== "agent" && folder.kind !== "project")) return; // not an agent/project home: contract says return, no tools + const home = homeChatFor(folder, chat.id); + installTools(pi, home); + hearOfficePings(pi, chat.id); // plan 5.4 step 4: the same factory owns the drain side of the wake queue +} + +/** The folder's home chat id (plan 2.1 -- meta.owner "is" the creating agent/project's home chat): the sessions working in the folder are its home chats, lowest id stands for them all (ping.ts' resolveMention rule). `fallback` covers a folder with no listed sessions -- the calling chat itself is in it by construction. */ +function homeChatFor(folder: ToolsFolderInfo, fallback: string): string { + const chats = [...inChats(folder.path)].sort((a, b) => a.id.localeCompare(b.id)); + return chats.length ? chats[0]!.id : fallback; +} + +/* ------------------------------------------------------------------ */ +/* refusals + tool plumbing (docs/tools.js idiom) */ +/* ------------------------------------------------------------------ */ + +/** A refusal the agent reads: it starts with its code, "error:<code> <message>" (the office contract's fix of docs' `<code>: <message>`). */ +class Refusal extends Error { + constructor(public readonly code: string, message: string) { + super(`error:${code} ${message}`); + } +} + +/** What a tool may throw out to the agent: refusals as-is, coded store/ooxml errors by their code (zip/part damage -> `corrupt`, revision_mismatch with the FRESH mtime in the message), anything else logged and generic. */ +function failed(e: unknown): unknown { + if (e instanceof Refusal) return e; + const err = e as (Error & { code?: string; currentMtimeMs?: number }) | null; + if (err && typeof err.message === "string" && typeof err.code === "string") { + if (err.code === "revision_mismatch") { + // The gate refused: tell the agent where the document really is so it can re-read and redo. + return new Refusal("revision_mismatch", `${err.message} Current revision: ${err.currentMtimeMs ?? "unknown"} -- office_read again, then redo the edit with that mtimeMs as baseRevision. Nothing was written.`); + } + const code = err.code === "invalid_zip" || err.code === "missing_part" ? "corrupt" : err.code; + return new Refusal(code, err.message); + } + logLine(`an office tool failed: ${err?.stack ?? e}`); + return new Refusal("failed", String(e)); +} + +/** `text` with a details side-channel, exactly docs' shape. */ +const text = (content: string, details?: unknown) => ({ content: [{ type: "text", text: content }], ...(details !== undefined ? { details } : {}) }); + +/** One office_* tool result must not blow the agent context on a huge grid or document; both office_read and + * office_read's sibling office_comments truncate at this length with an explicit marker (docs' shape). */ +const MAX_READ_CHARS = 200_000; + +/** The library file of a doc (docs/<docId>/<title>.<ext>), or a not_found refusal. */ +function readBytes(doc: DocMeta, root = rootDir()): Buffer { + const p = path.join(root, "docs", doc.docId, `${doc.title}.${doc.ext}`); + let buf: Buffer; + try { + buf = fs.readFileSync(p); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === "ENOENT") throw new Refusal("not_found", `the library file of "${doc.title}" is missing from disk`); + throw e; + } + return buf; +} + +/** The most editing humans, and whether the doc is locked right now, as one readable line ("" when nobody). */ +const lockLine = (doc: DocMeta) => { + const users = usersOf(doc.docId); + if (!users.length && !isLocked(doc.docId)) return ""; + const who = users.map((u) => u.name || u.id).join(", "); + return `\n${users.length ? `${users.length} person${users.length === 1 ? "" : "s"} editing now (${who}).` : ""}${isLocked(doc.docId) && !users.length ? "\nThe document is currently locked by the editor." : ""}`; +}; + +/* ------------------------------------------------------------------ */ +/* per-chat tools */ +/* ------------------------------------------------------------------ */ + +function installTools(pi: unknown, home: string): void { + const p = pi as any; // portal's extend factory hands a pi that carries registerTool (host mode) or not; docs' own tools use it untyped too. + + /** A doc in THIS home's scope by id -- owner === home OR granted to home (docsForHome); anything else is `not_found` with the SAME words as a nonexistent id, so another home's docs do not leak into view. */ + function visibleDoc(id: unknown): DocMeta { + if (typeof id !== "string" || !id) throw new Refusal("not_found", "There is no document with that id. Ids come from office_list and office_create"); + const doc = docsForHome(home, rootDir()).find((d) => d.docId === id); + if (!doc) throw new Refusal("not_found", "There is no document with that id. Ids come from office_list and office_create (a document of another agent/project is invisible to you until it mentions you in a comment)"); + return doc; + } + + const guarded = (work: (params: Record<string, unknown>) => Promise<unknown> | unknown) => async (_callId: unknown, params: Record<string, unknown>) => { + try { + return await work(params ?? {}); + } catch (e) { + throw failed(e); + } + }; + + p.registerTool({ + name: "office_list", + label: "List office documents", + description: "List the office documents of your library: every Word (.docx) and spreadsheet (.xlsx) document you own or were granted (by being mentioned in one of its comments). Each line has the id, title, format, owner, how many grants it holds, its current revision (mtimeMs -- pass that back as baseRevision to office_edit) and whether a human is editing it right now.", + promptSnippet: "List the office documents you own or were granted", + parameters: { + type: "object", + properties: {}, + }, + execute: guarded(() => { + const docs = docsForHome(home, rootDir()); + if (!docs.length) return text("No documents in your office library yet. Make one with office_create.", { action: "list", home, documents: [] }); + const lines = docs.map((doc) => { + const users = usersOf(doc.docId); + return `- ${doc.docId} "${doc.title}" ${doc.ext} owner ${doc.owner ?? "(ownerless)"} ${doc.grants.length} grant${doc.grants.length === 1 ? "" : "s"} revision ${doc.mtimeMs}${users.length ? ` ${users.length} editing now` : ""}`; + }); + return text(`${docs.length} document${docs.length === 1 ? "" : "s"}:\n${lines.join("\n")}`, { + action: "list", + home, + documents: docs.map((doc) => ({ id: doc.docId, title: doc.title, ext: doc.ext, owner: doc.owner, grants: doc.grants.length, sizeBytes: doc.sizeBytes, mtimeMs: doc.mtimeMs, locked: isLocked(doc.docId) || usersOf(doc.docId).length > 0 })), + }); + }), + }); + + p.registerTool({ + name: "office_create", + label: "Create an office document", + description: "Make a new blank office document in the shared library: type \"word\" (a .docx) or \"cell\" (an .xlsx sheet), both starting from a minimal template you edit with office_edit or open in the editor. It is owned by you, so every chat of your home sees it; other agents get access only when it mentions them in a comment. A title+type pair already taken is refused.", + promptSnippet: "Create a blank Word or spreadsheet document", + executionMode: "sequential", + parameters: { + type: "object", + properties: { + title: { type: "string", description: "The document's title (not empty)." }, + type: { type: "string", enum: ["word", "cell"], description: "\"word\" makes a .docx, \"cell\" an .xlsx spreadsheet." }, + }, + required: ["title", "type"], + }, + execute: guarded(async ({ title, type }) => { + const t = typeof title === "string" ? title.trim() : ""; + if (!t) throw new Refusal("invalid", "A document needs a title"); + if (type !== "word" && type !== "cell") throw new Refusal("unsupported_format", `type is \"word\" or \"cell\" (got ${typeof type === "string" ? `"${type}"` : String(type)})`); + const doc = await createDoc({ title: t, type, owner: home, createdBy: "agent" }, rootDir()); // throws {code:"exists"} on a duplicate title+type + record({ subject: doc.title, reason: `office_create: new ${doc.ext} document ${doc.docId}, owner ${home}`, chatId: home }); + return text(`Created "${doc.title}" (id ${doc.docId}, ${doc.ext}, revision ${doc.mtimeMs}). Edit it with office_edit, which needs that revision as baseRevision.`, { action: "create", id: doc.docId, title: doc.title, ext: doc.ext, mtimeMs: doc.mtimeMs, owner: doc.owner }); + }), + }); + + p.registerTool({ + name: "office_read", + label: "Read an office document", + description: "Read a document of your library by its id: the extracted text (paragraph-delimited for Word; a per-sheet grid of cell values for spreadsheets) plus, for Word, its comments (id, author, date, text). The revision line shows mtimeMs -- hand it to office_edit as baseRevision. It also says whether a human is editing right now.", + promptSnippet: "Read the text and comments of an office document by id", + parameters: { + type: "object", + properties: { + id: { type: "string", description: "The document's id, from office_list or office_create." }, + }, + required: ["id"], + }, + execute: guarded(async ({ id }) => { + const doc = visibleDoc(id); + const buf = readBytes(doc); + const lock = lockLine(doc); + if (doc.ext === "xlsx") { + // Plan 5.5: office_read returns a grid of cell values for xlsx; its COMMENTS are not in v1 (xl/commentsN.xml + VML, M4). + const parsed = parseXlsx(buf); // throws {code:"invalid_zip"} -> corrupt + const sheets = parsed.sheetNames.map((name) => { + const rows = parsed.sheets[name] ?? []; + return `Sheet "${name}" (${rows.length} row${rows.length === 1 ? "" : "s"}):\n${rows.map((r) => "| " + r.join(" | ") + " |").join("\n")}`; + }).join("\n\n"); + const truncatedSheets = Object.keys(parsed.truncatedSheets); + let said = `"${doc.title}" (id ${doc.docId}, xlsx, revision ${doc.mtimeMs})${lock}\n\n${sheets || "(empty workbook)"}\n\nNo comments: spreadsheet comments are not read in v1 (they arrive later).`; + if (truncatedSheets.length) said += `\n\n(truncation: sheet ${truncatedSheets.map((n) => `"${n}"`).join(", ")} referenced cells beyond the supported grid size; those far-out cells were not read)`; + if (said.length > MAX_READ_CHARS) said = said.slice(0, MAX_READ_CHARS) + "\n\n[output truncated by office_read: it exceeds 200,000 characters]"; + return text(said, { action: "read", id: doc.docId, ext: doc.ext, mtimeMs: doc.mtimeMs, sheets: parsed.sheetNames }); + } + const parsed = parseDocx(buf); // throws {code:"invalid_zip"|"missing_part"} -> corrupt + let comments = ""; + if (parsed.comments.length) { + const lines = parsed.comments.map((c) => `- ${c.id} ${c.author}${c.date ? ` ${c.date}` : ""}: ${c.text.replace(/\n/g, " ")}`); + comments = `\n\nComments (${parsed.comments.length}):\n${lines.join("\n")}`; + } + let said = `"${doc.title}" (id ${doc.docId}, docx, revision ${doc.mtimeMs})${lock}\n\n${parsed.text}${comments}`; + if (said.length > MAX_READ_CHARS) said = said.slice(0, MAX_READ_CHARS) + "\n\n[output truncated by office_read: it exceeds 200,000 characters]"; + // details lands in the portal tool-event store and the web transcript -- MAX_READ_CHARS caps only the text above, + // so bound the comments HERE too (100 at 500 chars each, worst case ~50 KB); commentCount keeps the true total. + const detailComments = parsed.comments.slice(0, 100).map((c) => ({ id: c.id, author: c.author, date: c.date, text: c.text.slice(0, 500) })); + return text(said, { action: "read", id: doc.docId, ext: doc.ext, mtimeMs: doc.mtimeMs, comments: detailComments, commentCount: parsed.comments.length }); + }), + }); + + p.registerTool({ + name: "office_edit", + label: "Edit an office document", + description: `Change a Word document of your library by replacing quoted text, as office_read shows it. Each op is {quote, replacement}: the quote must appear in the extracted text (empty replacement removes it), ops apply one after another. You are a CO-EDITOR alongside any human who has the document open -- there is no session check and no switch; the edit always proceeds under the revision gate: baseRevision is the mtimeMs you read (or that your last edit returned), and if the document moved on since (for example a co-editor's save) NOTHING is written and you hear a CONFLICT with the fresh revision -- re-read and retry. You never clobber a newer revision. Spreadsheets cannot be edited in v1 (a later milestone). All ops are made or none.`, + promptSnippet: "Edit a Word document: quoted replacements on the revision you read", + executionMode: "sequential", + parameters: { + type: "object", + properties: { + id: { type: "string", description: "The document's id, from office_list or office_create." }, + ops: { + type: "array", + minItems: 1, + items: { + type: "object", + properties: { + quote: { type: "string", description: "Text to replace, exactly as office_read shows it (may not be empty)." }, + replacement: { type: "string", description: "What takes its place; empty removes the quote." }, + }, + required: ["quote", "replacement"], + }, + }, + baseRevision: { type: "number", description: "The revision (mtimeMs) you read or last edited at." }, + }, + required: ["id", "ops", "baseRevision"], + }, + execute: guarded(async ({ id, ops, baseRevision }) => { + const doc = visibleDoc(id); + if (doc.ext !== "docx") throw new Refusal("unsupported_format", `"${doc.title}" is an .xlsx spreadsheet: editing spreadsheets is not in v1 (a later milestone) -- read it with office_read`); + if (!Array.isArray(ops) || ops.length === 0) throw new Refusal("invalid", "ops is a non-empty list of {quote, replacement}"); + ops.forEach((op, i) => { + if (typeof op?.quote !== "string" || !op.quote || typeof op.replacement !== "string") throw new Refusal("invalid", `op ${i + 1} needs a quote (not empty) and a replacement (text, or empty to remove)`); + }); + const base = Number(baseRevision); + if (baseRevision === undefined || baseRevision === null || !Number.isFinite(base)) throw new Refusal("invalid", "baseRevision is the mtimeMs revision office_list/office_read showed you"); + + // GATE FIRST (contract): the meta as it stands now vs what the agent read; >1 ms drift means a save + // moved since -- refuse with the FRESH mtime in the message, and write nothing. store.persistBytes + // re-gates strictly at write time too, so this is belt and not only braces. + let meta = getDoc(doc.docId, rootDir()); + if (!meta) throw new Refusal("not_found", `"${doc.title}" vanished from the library between list and edit`); + // The co-editor model: NO session check here (live.ts presence neither allows nor blocks) -- the revision + // gate alone decides. When it refuses, say LOUDLY that this is a CONFLICT, name any open human editor so + // the agent learns somebody is co-editing, and point at the fresh revision to re-read + retry. + const stale = (fresh: DocMeta) => { + const humans = usersOf(fresh.docId); + const openEditorNote = humans.length > 0 ? ` A human editor (${humans.map((h) => h.name ?? h.id).join(", ")}) still has the document open, so a co-editor is likely behind the change.` : ""; + return new Refusal("revision_mismatch", `CONFLICT: "${fresh.title}" is at revision ${fresh.mtimeMs}, not ${base}: it changed since you read it. Nothing was written.${openEditorNote} Read it again with office_read, then redo the edit with baseRevision ${fresh.mtimeMs}.`); + }; + if (Math.abs(meta.mtimeMs - base) > 1) throw stale(meta); + + // word/document.xml out of the archive; damage is `corrupt`, quote problems pass through ooxml's codes. + const buf = readBytes(doc); + let documentXml: string; + try { + const zip = await (await jsZip()).loadAsync(buf); + const part = zip.file("word/document.xml"); + if (!part) throw new Refusal("corrupt", `"${doc.title}" has no word/document.xml: it is not a Word document`); + documentXml = (await part.async("string")) ?? ""; + } catch (e) { + if (e instanceof Refusal) throw e; + throw new Refusal("corrupt", `"${doc.title}" could not be opened as a zip archive: ${e instanceof Error ? e.message : String(e)}`); + } + + const editedXml = editDocx(documentXml, ops as EditOp[]); // throws {code:"quote_not_found"|"quote_ambiguous"} -- surfaced unchanged by failed() + const out = rewriteZip(buf, { "word/document.xml": editedXml }); // {code:"invalid_zip"} -> corrupt + let saved: { mtimeMs: number }; + try { + saved = persistBytes(meta, out, rootDir()); // the strict ZERO-tolerance final gate + index update + } catch (e) { + const err = e as (Error & { code?: string; currentMtimeMs?: number }) | null; + if (err?.code === "revision_mismatch") { + // The file moved underneath us between the pre-gate read and the write (a co-editor's save landing + // on disk through the DS callback persist path). Refuse LOUD as a conflict with the FRESH revision, + // generic wording here: this is the safety mechanism of the co-editor model (fail loud, re-read + retry). + const humans = usersOf(doc.docId); + const note = humans.length > 0 ? ` A human editor (${humans.map((h) => h.name ?? h.id).join(", ")}) has the document open -- a co-editor's save is the likely change.` : ""; + throw new Refusal("revision_mismatch", `CONFLICT: "${doc.title}" changed while this edit was in flight: it is now at revision ${err.currentMtimeMs ?? "unknown"}, you based the edit on ${base}. Nothing was written.${note} Read it again with office_read, then redo the edit with baseRevision ${err.currentMtimeMs ?? "the fresh revision office_list shows"}.`); + } + throw e; + } + record({ subject: doc.title, reason: `office_edit: ${ops.length} op(s) applied, revision ${meta.mtimeMs} -> ${saved.mtimeMs}`, chatId: home }); + const n = ops.length; + return text(`Edited "${doc.title}" (${n} op${n === 1 ? "" : "s"} made): now at revision ${saved.mtimeMs}. Pass that as baseRevision for the next edit.`, { action: "edit", id: doc.docId, from: meta.mtimeMs, to: saved.mtimeMs, ops: n }); + }), + }); + + p.registerTool({ + name: "office_comments", + label: "List document comments and grants", + description: "Read-only: the comments of a Word document (id, author, date, text) plus its grants -- which other agent/project homes hold access to it by mention. Spreadsheets have no comment reading in v1.", + promptSnippet: "List the comments and grants of an office document", + parameters: { + type: "object", + properties: { + id: { type: "string", description: "The document's id, from office_list or office_create." }, + }, + required: ["id"], + }, + execute: guarded(async ({ id }) => { + const doc = visibleDoc(id); + if (doc.ext === "xlsx") throw new Refusal("unsupported_format", `"${doc.title}" is an .xlsx spreadsheet: its comments live in a different part layout and are not read in v1 (they arrive in M4)`); + const parsed = parseDocx(readBytes(doc)); // corrupt mapping via failed() + const commentLines = parsed.comments.length ? parsed.comments.map((c) => `- ${c.id} ${c.author}${c.date ? ` ${c.date}` : ""}: ${c.text.replace(/\n/g, " ")}`).join("\n") : "(none)"; + const grantLines = doc.grants.length ? doc.grants.map((g) => `- ${g.chatId} via comment ${g.commentId} at ${new Date(g.at).toISOString()}`).join("\n") : "(none -- only the owner can see this document)"; + let said = `"${doc.title}" (id ${doc.docId})\n\nComments (${parsed.comments.length}):\n${commentLines}\n\nGrants:\n${grantLines}`; + // Same cap and marker as office_read: aggregated comment text is unbounded input and must not blow the agent context. + if (said.length > MAX_READ_CHARS) said = said.slice(0, MAX_READ_CHARS) + "\n\n[output truncated by office_comments: it exceeds 200,000 characters]"; + // details lands in the portal tool-event store and the web transcript -- bound it like office_read does + // (100 at 500 chars each, worst case ~50 KB); commentCount keeps the true total. + const detailComments = parsed.comments.slice(0, 100).map((c) => ({ id: c.id, author: c.author, date: c.date, text: c.text.slice(0, 500) })); + return text(said, { action: "comments", id: doc.docId, comments: detailComments, grants: doc.grants, commentCount: parsed.comments.length }); + }), + }); +} + +/** + * The drain side of plan 5.4 step 4 (docs' hearPings): on the UI_RECEIVE bell with data.type "ping" + * -- which ping.ts rings per woken home via chats.tell(chatId, {type:"ping"}) -- take this chat's queued + * wake entries off ping.ts' server-side queue and hand each to pi: shown to the person, delivered as a + * follow-up that triggers a turn, carrying {docId, commentId} in details. A reloaded pi keeps the bus: + * the handler of the one that went is removed on session shutdown (docs does exactly this). + */ +function hearOfficePings(pi: unknown, chatId: string): void { + const p = pi as any; + if (typeof p?.events?.on !== "function") return; // no event bus in this pi build: the tools still work, wakes just wait for a drain by another means + const off = p.events.on(UI_RECEIVE, (message: { data?: unknown }) => { + if ((message as { data?: { type?: string } })?.data?.type !== "ping") return; // ours is data.type "ping" (ping.ts sets it) + for (const entry of drainWakeQueue(chatId)) { + try { + p.sendMessage({ customType: "office-comment-ping", content: entry.text, display: true, details: entry.details }, { deliverAs: "followUp", triggerTurn: true }); + } catch (e) { + logLine(`could not deliver a queued office wake to chat ${chatId}: ${(e as Error)?.message ?? e}`); + } + } + }); + p.on?.("session_shutdown", () => off?.()); +} diff --git a/extensions/office/src/web/api.ts b/extensions/office/src/web/api.ts new file mode 100644 index 000000000..ee0f23467 --- /dev/null +++ b/extensions/office/src/web/api.ts @@ -0,0 +1,95 @@ +import { portal } from "../portal"; + +/** + * What the browser half asks of its server half: the routes under the address the portal gives the package + * (portal.api.url, core-mounted at /api/x/pithagoras-office, session-gated there). A request made while the + * server half is not running is refused as one the server refused is. + */ + +/** One document of the library, as GET /state says it (store.DocMeta verbatim). */ +export interface OfficeDoc { + docId: string; + title: string; + ext: "docx" | "xlsx"; + createdAt: number; + updatedAt: number; + sizeBytes: number; + /** The document's current revision: its file's mtimeMs, what office_edit passes back as baseRevision. */ + mtimeMs: number; + /** The home chat that created it, or null for one made from the global page. */ + owner: string | null; + /** Mentions in its comments that opened it to those homes. */ + grants: { chatId: string; commentId: string; at: number }[]; + createdBy?: string; + /** Which workspace dir the doc lives in, stamped by the scan (store.ts); ABSENT for library-made docs -- the UI shows "library". */ + workspace?: string; +} + +/** A pull in progress, out of the server half's memory (container.PullState). */ +export interface OfficePull { + active: boolean; + line: string; + error?: string; +} + +/** The Document Server container, as status() reports it. */ +export interface OfficeContainer { + state: string; + version?: string; + memBytes?: number; + pull?: OfficePull | null; +} + +/** GET /state -- the payload of the 5 s poll (plan 5.7 tier 1): settings, container, every doc, who is in what. */ +export interface OfficeState { + enabled: boolean; + imageTag: string; + container: OfficeContainer; + docs: OfficeDoc[]; + /** Every doc has an entry: empty while nobody is editing it, so the badge code needs no null check. */ + locks: Record<string, { id: string; name?: string }[]>; +} + +/** POST /editor-config: the plain DocsAPI.DocEditor config plus its JWT and where api.js lives -- apiJs is dropped again before the call. */ +export interface EditorConfig extends Record<string, unknown> { + token: string; + apiJs: string; +} + +export type SettingsPatch = { enabled?: boolean; imageTag?: string }; + +/** The settings answer: what was saved, where the container stands as of now, and that a start went on its own. */ +export interface SettingsAnswer { + settings: { enabled: boolean; imageTag: string }; + container: OfficeContainer; + starting?: boolean; +} + +/** A JSON route of the server half, at its address. */ +const call = async <T>(path: string, init?: RequestInit): Promise<T> => portal.api.json<T>(portal.api.url(path), init); + +export const api = { + state: () => call<OfficeState>("/state"), + createDocument: (body: { title: string; type: "word" | "cell"; owner?: string }) => + call<OfficeDoc>("/documents", { method: "POST", body: JSON.stringify(body) }), + renameDocument: (id: string, title: string) => + call<OfficeDoc>(`/documents/${encodeURIComponent(id)}/rename`, { method: "POST", body: JSON.stringify({ title }) }), + deleteDocument: (id: string) => call<{ ok: true }>(`/documents/${encodeURIComponent(id)}`, { method: "DELETE" }), + /** Re-run the workspace scan now (boot does it automatically): { added, updated }. */ + rescanLibrary: () => call<{ added: number; updated: number }>("/documents/rescan", { method: "POST" }), + editorConfig: (docId: string) => call<EditorConfig>("/editor-config", { method: "POST", body: JSON.stringify({ doc: docId }) }), + saveSettings: (patch: SettingsPatch) => call<SettingsAnswer>("/settings", { method: "POST", body: JSON.stringify(patch) }), + /** Awaits the whole first pull to completion -- the page shows its progress from /state meanwhile. */ + startContainer: () => call<{ started: boolean; container: OfficeContainer }>("/container/start", { method: "POST" }), + stopContainer: () => call<{ container: OfficeContainer }>("/container/stop", { method: "POST" }), + /** The library bytes as an attachment -- plain addresses, the session rides along on the cookie. */ + downloadUrl: (id: string) => portal.api.url(`/documents/${encodeURIComponent(id)}`), + exportUrl: (id: string) => portal.api.url(`/documents/${encodeURIComponent(id)}/export`), +}; + +/** A refusal of the server half, in words for where it says the rest. The ApiError of core's json() carries + * exactly that message (the { error } field), so a failure of POST /documents -- including the retryable 503 + * "dependencies_pending" while jszip is still installing -- reaches an ErrorBanner verbatim, never a dead click. */ +export function sayError(e: unknown): string { + return e instanceof Error ? e.message : String(e); +} diff --git a/extensions/office/src/web/library.tsx b/extensions/office/src/web/library.tsx new file mode 100644 index 000000000..2c0ef72d6 --- /dev/null +++ b/extensions/office/src/web/library.tsx @@ -0,0 +1,359 @@ +import { useEffect, useMemo, useState } from "react"; +import { LuDownload, LuFileText, LuPresentation, LuRefreshCw, LuTable, LuTrash2, LuUsers, LuUpload } from "react-icons/lu"; +import { officeUrl, portal, t } from "../portal"; +import { compareDocs, loadSort, saveSort, SortBar, workspaceLabel, type SortChoice } from "./sort"; +import { api, sayError, type OfficeDoc, type OfficeState } from "./api"; +import { editorsIn } from "../office-ui"; +import { parseDocQuery } from "../office-ui"; +import { OfficeEditor } from "./office-editor"; +import { refreshOfficeState, useOfficeState } from "./state"; + +/** + * The Office page -- the global tab's view of the library. With `?doc=<id>` in its address it is the full-bleed + * editor (the docs extension keeps its opened document in the page the same way); without it, the workbench: + * a new-document line and the list of every document with who owns it and who is in it. The list itself is the + * workspace scan -- every .docx/.xlsx of the agent homes and projects, indexed at boot and on Rescan, each in + * place (its file IS the document) -- plus what was made here. + */ + +// How the list sorts and where that choice sticks live in web/sort.tsx, shared with the sidebar panel: +// one of five keys (name / modified / size / type / workspace), one direction, persisted under "po-sort". + +function extMark(doc: OfficeDoc) { + return doc.ext === "docx" ? ( + <LuFileText className="po-ext po-ext-word" aria-hidden /> + ) : ( + <LuTable className="po-ext po-ext-cell" aria-hidden /> + ); +} + +/** How the document's mtimeMs reads: when it last changed on disk, which IS its revision. */ +const when = (doc: OfficeDoc) => portal.format.dateTime(doc.mtimeMs); + +function ownerChips(doc: OfficeDoc) { + return ( + <> + {doc.owner ? ( + <span className="po-chip po-chip-owner" title={t("Owned by this agent/project's home")}> + {t("owned")} · {doc.owner} + </span> + ) : ( + <span className="po-chip">{t("unowned")}</span> + )} + {doc.grants.length > 0 && ( + <span className="po-chip" title={doc.grants.map((g) => g.chatId).join(", ")}> + {doc.grants.length === 1 ? t("1 granted") : t("{n} granted", { n: doc.grants.length })} + </span> + )} + </> + ); +} + +/** The document's editing badge, straight out of the polled /state: every doc has a locks entry. */ +function lockChip(state: OfficeState, doc: OfficeDoc) { + const n = editorsIn(state.locks, doc.docId); + return n > 0 ? ( + <span className="po-chip po-chip-lock"> + <LuUsers aria-hidden /> + {n === 1 ? t("1 editing now") : t("{n} editing now", { n })} + </span> + ) : null; +} + +/** + * One document in the library, with what can be done to it: opened (the row's title), downloaded or exported as a + * file (plain addresses -- the session rides along), renamed in place (a form takes the row's title), deleted after + * the portal's question, which says where it goes: the office trash, swept after 7 days. The action buttons are out + * of the way until the row is pointed at or has focus (and always there on a screen without pointing). + */ +function DocRow({ doc }: { doc: OfficeDoc }) { + portal.locale.use(); + const state = useOfficeState(); + const [renaming, setRenaming] = useState(false); + const [title, setTitle] = useState(doc.title); + const [busy, setBusy] = useState(false); + const [problem, setProblem] = useState<string | null>(null); + + useEffect(() => setTitle(doc.title), [doc.title]); + + const saveRename = async () => { + const next = title.trim(); + if (next === "" || next === doc.title) return setRenaming(false); + setBusy(true); + setProblem(null); + try { + await api.renameDocument(doc.docId, next); + setRenaming(false); + void refreshOfficeState(); + } catch (e) { + setProblem(sayError(e)); + } finally { + setBusy(false); + } + }; + + const remove = async () => { + if (busy) return; + const ok = await portal.ui.confirm({ + title: t("Delete “{title}”?", { title: doc.title }), + message: t("The document is moved to the office trash, which is swept after 7 days. Its revisions and history go with it."), + confirmLabel: t("Delete"), + danger: true, + deletes: true, + }); + if (!ok) return; + setBusy(true); + setProblem(null); + try { + await api.deleteDocument(doc.docId); + void refreshOfficeState(); + } catch (e) { + setProblem(sayError(e)); + } finally { + setBusy(false); + } + }; + + return ( + <li className="po-row"> + {renaming ? ( + <form + className="po-form" + style={{ flex: 1, border: "none", padding: 0 }} + onSubmit={(e) => { + e.preventDefault(); + void saveRename(); + }} + > + <input + className={`po-grow ${portal.ui.classes.input}`} + value={title} + maxLength={200} + aria-label={t("New title for {title}", { title: doc.title })} + onChange={(e) => setTitle(e.target.value)} + /> + <button type="submit" className={portal.ui.classes.primarySmall} disabled={busy || title.trim() === ""}> + {t("Save")} + </button> + <button type="button" className={portal.ui.classes.ghost} onClick={() => setRenaming(false)}> + {t("Cancel")} + </button> + </form> + ) : ( + <button type="button" className="po-open" onClick={() => portal.navigate(officeUrl(doc.docId))}> + {extMark(doc)} + <span style={{ minWidth: 0 }}> + <span className="po-title">{doc.title}</span> + <span className="po-meta"> + {t("{title}.{ext}", { title: doc.title, ext: doc.ext })} · {when(doc)} · {workspaceLabel(doc)} + </span> + </span> + </button> + )} + {state && !renaming && ( + <span className="po-chips"> + {ownerChips(doc)} + {lockChip(state, doc)} + </span> + )} + {!renaming && ( + <span className="po-row-actions"> + <a className={portal.ui.classes.iconButton} href={api.downloadUrl(doc.docId)} title={t("Download {title}", { title: doc.title })} aria-label={t("Download {title}", { title: doc.title })}> + <LuDownload aria-hidden /> + </a> + <a className={portal.ui.classes.iconButton} href={api.exportUrl(doc.docId)} title={t("Export a fresh copy")} aria-label={t("Export a fresh copy")}> + <LuUpload aria-hidden /> + </a> + <button + type="button" + className={portal.ui.classes.iconButton} + title={t("Rename {title}", { title: doc.title })} + aria-label={t("Rename {title}", { title: doc.title })} + onClick={() => setRenaming(true)} + > + ✎ + </button> + <button + type="button" + className={portal.ui.classes.iconButton} + aria-disabled={busy} + title={t("Delete {title}", { title: doc.title })} + aria-label={t("Delete {title}", { title: doc.title })} + onClick={() => void remove()} + > + <LuTrash2 aria-hidden /> + </button> + </span> + )} + {problem && ( + <portal.ui.ErrorBanner onClose={() => setProblem(null)}>{t("That did not work: {error}", { error: problem })}</portal.ui.ErrorBanner> + )} + </li> + ); +} + +/** A new document in the library: its title and which kind -- word (.docx) or sheet (.xlsx). Human-made ones are unowned. */ +function NewDocument() { + portal.locale.use(); + const [title, setTitle] = useState(""); + const [kind, setKind] = useState<"word" | "cell">("word"); + const [busy, setBusy] = useState(false); + const [problem, setProblem] = useState<string | null>(null); + const make = async () => { + if (title.trim() === "" || busy) return; + setBusy(true); + setProblem(null); + try { + await api.createDocument({ title: title.trim(), type: kind }); + setTitle(""); + void refreshOfficeState(); + } catch (e) { + setProblem(sayError(e)); + } finally { + setBusy(false); + } + }; + return ( + <form + className="po-form" + onSubmit={(e) => { + e.preventDefault(); + void make(); + }} + > + <input + className={`po-grow ${portal.ui.classes.input}`} + value={title} + maxLength={200} + placeholder={t("Title of a new document")} + aria-label={t("Title of a new document")} + onChange={(e) => setTitle(e.target.value)} + /> + <portal.ui.Segments + label={t("Kind")} + value={kind} + options={[ + { id: "word", label: t("Word (.docx)") }, + { id: "cell", label: t("Sheet (.xlsx)") }, + ]} + onChange={setKind} + showLabel={false} + /> + <button type="submit" className={portal.ui.classes.primarySmall} disabled={busy || title.trim() === ""}> + {t("New document")} + </button> + {problem && ( + <portal.ui.ErrorBanner onClose={() => setProblem(null)}>{t("The document could not be made: {error}", { error: problem })}</portal.ui.ErrorBanner> + )} + </form> + ); +} + +/** Re-run the workspace scan from the page: boot already scanned once, this catches files written straight to + * disk since (an agent or a person working in the homes and projects) without waiting for the next portal start. + * The answer's counts are said out loud in a hint line, not an alert. */ +function RescanLibrary() { + portal.locale.use(); + const [busy, setBusy] = useState(false); + const [note, setNote] = useState<string | null>(null); + const go = async () => { + if (busy) return; + setBusy(true); + setNote(null); + try { + const out = await api.rescanLibrary(); + setNote( + out.added === 0 && out.updated === 0 + ? t("Nothing new: the library already knew every .docx and .xlsx in the agent folders.") + : t("Scan done: {added} indexed, {updated} refreshed.", { added: out.added, updated: out.updated }), + ); + void refreshOfficeState(); + } catch (e) { + setNote(t("The rescan did not work: {error}", { error: sayError(e) })); + } finally { + setBusy(false); + } + }; + return ( + <span className="po-rescan"> + <button type="button" className={portal.ui.classes.ghost} onClick={() => void go()} disabled={busy} aria-label={t("Rescan the agent folders for .docx and .xlsx files")}> + <LuRefreshCw aria-hidden /> {t("Rescan workspace")} + </button> + {note && <span className="po-hint">{note}</span>} + </span> + ); +} + +// The SortBar itself (key incl. workspace + direction, "po-sort" persistence) is the shared one from web/sort.tsx. + +function Library() { + portal.locale.use(); + const state = useOfficeState(); + const [sort, setSort] = useState(loadSort); // shared with the sidebar panel ("po-sort"); default modified-descending + const chooseSort = (next: SortChoice) => { + setSort(next); + saveSort(next); + }; + const docs = useMemo( + () => (state ? [...state.docs].sort((a, b) => compareDocs(a, b, sort.key, sort.dir)) : []), + [state, sort], + ); // client-side ordering of the polled payload; the server hands back raw meta + return ( + <div className="po-page"> + <portal.ui.PageHeader + icon={<LuPresentation />} + title={t("Office")} + description={t("The office documents of this portal: everything made here plus every .docx or .xlsx found across the agent homes and projects -- indexed automatically at boot and by Rescan, editable in place.")} + /> + {!state ? ( + <portal.ui.RowsSkeleton rows={4} label={t("Loading…")} /> + ) : ( + <> + <NewDocument /> + {state.docs.length === 0 ? ( + <> + <portal.ui.Empty>{t("No documents yet. Make a Word or sheet document above, ask an agent to write one -- or drop a .docx / .xlsx into any agent folder and press Rescan: everything is indexed here automatically, in place.")}</portal.ui.Empty> + <RescanLibrary /> + </> + ) : ( + <> + <span className="po-listhead"> + <SortBar sort={sort} onSort={chooseSort} /> + <RescanLibrary /> + </span> + <ul className="po-list"> + {docs.map((doc) => ( + <DocRow key={doc.docId} doc={doc} /> + ))} + </ul> + </> + )} + </> + )} + </div> + ); +} + +/** The page itself: the document named by `?doc=` in full bleed, else the library (the docs extension's precedent). */ +export function OfficePage() { + const { search } = portal.useLocation(); + const docId = parseDocQuery(search); + const state = useOfficeState(); + if (docId) { + // The document is gone from the library meanwhile: say so, with a way back. + if (state && !state.docs.some((d) => d.docId === docId)) { + return ( + <div className="po-page"> + <portal.ui.ErrorBanner> + {t("This document is no longer in the library.")}{" "} + <button type="button" className={portal.ui.classes.ghost} onClick={() => portal.navigate(officeUrl())}> + ← {t("All documents")} + </button> + </portal.ui.ErrorBanner> + </div> + ); + } + return <OfficeEditor docId={docId} onBack={() => portal.navigate(officeUrl())} />; + } + return <Library />; +} diff --git a/extensions/office/src/web/office-editor.tsx b/extensions/office/src/web/office-editor.tsx new file mode 100644 index 000000000..5784a9067 --- /dev/null +++ b/extensions/office/src/web/office-editor.tsx @@ -0,0 +1,160 @@ +import { useEffect, useRef, useState } from "react"; +import { LuUsers } from "react-icons/lu"; +import { portal, t } from "../portal"; +import { api, sayError } from "./api"; +import { useOfficeState } from "./state"; + +/** + * The Office editor's host: the page's half of `new DocsAPI.DocEditor`. It asks POST /editor-config for its + * document, loads the Document Server's api.js ONCE (the script is not idempotent -- a second load on the same + * page breaks it), drops ONLY apiJs from the answered config and hands the rest to DocsAPI as-is: the JWT covers + * exactly that object (plan 5.6 signs what is sent, so nothing is added to it). The lock badge ("N editing now") + * and the "agent updated this document" badge come from the shared 5 s poll of /state -- plan 5.7 tier 1; the + * real refreshFile handshake that would replace the page refresh is deferred (M0/M4). + */ + +/** What DocsAPI leaves on window once api.js has loaded: only what is used, typed as it behaves. */ +interface DocsApiLike { + DocEditor: new (placeholderId: string, config: Record<string, unknown>) => DestroyableEditor; +} +interface DestroyableEditor { + destroyEditor?: () => void; +} + +declare global { + interface Window { + DocsAPI?: DocsApiLike; + } +} + +/** api.js per address, loaded once per page: the promise is cached, and a failure stays said until the page reloads. */ +const scripts = new Map<string, Promise<void>>(); +function loadScript(src: string): Promise<void> { + const known = scripts.get(src); + if (known) return known; + const load = new Promise<void>((resolve, reject) => { + const el = document.createElement("script"); + el.src = src; + el.onload = () => resolve(); + el.onerror = () => reject(new Error(`the editor script at ${src} could not be loaded`)); + document.head.appendChild(el); + }); + scripts.set(src, load); + return load; +} + +export function OfficeEditor({ docId, onBack }: { docId: string; onBack(): void }) { + portal.locale.use(); + const state = useOfficeState(); + const doc = state?.docs.find((d) => d.docId === docId); + const [problem, setProblem] = useState<string | null>(null); + /** The Document Server has filled its iframe into the placeholder: the loading state is done. */ + const [ready, setReady] = useState(false); + const box = useRef<HTMLDivElement>(null); + const editor = useRef<DestroyableEditor | null>(null); + // What mtimeMs was in effect when this view opened (the revision the minted key carries, as near as the page + // knows it -- /state already had the doc by the time it could be opened): a poll that is newer than it means + // an agent edited meanwhile, which tier 1 says with the badge below. + const openedAt = useRef<number | null>(null); + + useEffect(() => { + if (openedAt.current === null && doc) openedAt.current = doc.mtimeMs; + }, [doc]); + + useEffect(() => { + let alive = true; + let watch: ReturnType<typeof setInterval> | undefined; + setProblem(null); + setReady(false); + void (async () => { + try { + const minted = await api.editorConfig(docId); + if (!alive || !box.current) return; + await loadScript(minted.apiJs); + if (!alive || !box.current) return; + // The JWT was signed over the config WITHOUT these two fields: drop exactly apiJs, send the rest unaltered. + const config: Record<string, unknown> = { ...minted }; + delete config.apiJs; + if (!window.DocsAPI) throw new Error("api.js loaded but DocsAPI is not on window -- the Document Server answered with something that is not its script"); + // DocsAPI.DocEditor takes a PLACEHOLDER ID: api.js runs document.getElementById on the first argument and then + // replaces that element with its iframe. Pass the host box itself and getElementById stringifies it to + // "[object HTMLDivElement]", finds nothing, and silently builds no editor (reproduced live against DS 10.0). + const ph = document.createElement("div"); + ph.id = "po-doceditor"; + box.current.querySelector("#po-doceditor")?.remove(); // never stack placeholders across remounts + box.current.appendChild(ph); + editor.current = new window.DocsAPI.DocEditor(ph.id, config); + // onDocumentReady, as close to it as this tier can get: DS fills its own iframe into the placeholder. + watch = setInterval(() => { + if (box.current?.querySelector("iframe")) { + setReady(true); + if (watch !== undefined) clearInterval(watch); + } + }, 500); + } catch (e) { + if (alive) setProblem(sayError(e)); + } + })(); + return () => { + alive = false; + if (watch !== undefined) clearInterval(watch); + try { + editor.current?.destroyEditor?.(); // where the loaded api.js exposes it + } catch { + // a destroyed or half-built editor must not take the page down with it + } + document.getElementById("po-doceditor")?.remove(); // sweep a placeholder DS never replaced; ours to clean, not React's + editor.current = null; + }; + }, [docId]); + + const users = state?.locks[docId] ?? []; + // An agent wrote a newer revision than the one this view opened on (plan 5.7 tier-1 badge). + const stale = openedAt.current !== null && !!doc && doc.mtimeMs > openedAt.current; + + return ( + <div className="po-page po-editor po-editor-page"> + <div className="po-editor-bar"> + <button type="button" className={portal.ui.classes.ghost} onClick={onBack}> + ← {t("All documents")} + </button> + <span className="po-editor-title">{doc ? `${doc.title}.${doc.ext}` : "…"}</span> + <span className="po-editor-spacer" /> + {users.length > 0 && ( + <span className="po-chip po-chip-lock" title={users.map((u) => u.name ?? u.id).join(", ")}> + <LuUsers aria-hidden /> + {users.length === 1 ? t("1 editing now") : t("{n} editing now", { n: users.length })} + </span> + )} + {stale && ( + <span className="po-chip po-chip-stale"> + {t("The agent updated this document")} + {" — "} + <button type="button" className="po-refresh" onClick={() => window.location.reload()}> + {t("refresh to see their changes")} + </button> + </span> + )} + </div> + {problem && ( + <portal.ui.ErrorBanner> + {problem}{" "} + <button + type="button" + className={portal.ui.classes.ghost} + onClick={() => portal.navigate(`/settings/x/${portal.extension}`)} + > + {t("Open the Office settings")} + </button> + </portal.ui.ErrorBanner> + )} + <div className="po-host" ref={box}> + {!ready && !problem && ( + <div className="po-loading"> + {t("Loading the editor — the first start of a Document Server can take a while…")} + </div> + )} + </div> + </div> + ); +} diff --git a/extensions/office/src/web/settings.tsx b/extensions/office/src/web/settings.tsx new file mode 100644 index 000000000..dcf1baadb --- /dev/null +++ b/extensions/office/src/web/settings.tsx @@ -0,0 +1,145 @@ +import { useState } from "react"; +import { portal, t } from "../portal"; +import { api, sayError } from "./api"; +import { floatingTag } from "../office-ui"; +import { refreshOfficeState, useOfficeState } from "./state"; + +/** + * The Office page in Settings: a service panel for the Document Server -- where it stands (and what a first pull is + * doing), start and stop, the pinned image tag, and where the documents live. There is no co-editing switch: the + * agent is a co-editor under the revision gate (fail loud, re-read + retry) and nothing on this page changes that. + * Enabling goes through + * POST /settings, which answers without waiting for a first pull to finish (`starting`): the pull progress keeps + * coming in over the shared 5 s poll, and the server half's lookAfter stays quiet while the person has stopped it. + */ + +function mem(bytes: number): string { + return bytes >= 2 ** 30 ? `${(bytes / 2 ** 30).toFixed(1)} GB` : `${Math.round(bytes / 2 ** 20)} MB`; +} + +export function OfficeSettings() { + portal.locale.use(); + const state = useOfficeState(); + const [busy, setBusy] = useState<null | "start" | "stop" | "settings">(null); + const [problem, setProblem] = useState<string | null>(null); + // Null: still saying what the server says. A string: what is in the input now. + const [tag, setTag] = useState<string | null>(null); + + if (!state) return <portal.ui.RowsSkeleton rows={4} label={t("Loading…")} />; + + const c = state.container; + const running = c.state === "running"; + const pull = c.pull; + + async function act(kind: NonNullable<typeof busy>, work: () => Promise<unknown>): Promise<void> { + setBusy(kind); + setProblem(null); + try { + await work(); + await refreshOfficeState(); + } catch (e) { + setProblem(sayError(e)); + } finally { + setBusy(null); + } + } + + const saveTag = () => + act("settings", async () => { + const next = (tag ?? state.imageTag).trim(); + if (next === "") throw new Error(t("{ imageTag } must be a non-empty string like onlyoffice/documentserver:10.0")); + // The same refusal the server makes, made here first: never a floating tag (it breaks the co-editing keys). + if (floatingTag(next)) throw new Error(t("Never pin :latest — a moved image changes the Document Server under you and breaks the co-editing keys.")); + await api.saveSettings({ imageTag: next }); + }); + + const start = () => + act("start", async () => { + // A first start pulls the whole image and takes minutes; the route awaits it, so this fetch stays open while + // the pull progress is shown from /state meanwhile. + await api.startContainer(); + setTag(null); + }); + + const stop = () => act("stop", async () => { + await api.stopContainer(); + setTag(null); + }); + + return ( + <div className="space-y-4"> + {/* The service itself: where it stands, and what a pull is doing. */} + <div className="po-service" role="status"> + <div className="po-service-head"> + <span + className={ + "h-2 w-2 rounded-full " + + (running ? "po-state-live bg-ok" : c.state === "starting" || (pull?.active ?? false) ? "po-state-pulling bg-warn" : "bg-fg-subtle") + } + /> + {t("OnlyOffice Document Server")} + </div> + <div className="po-service-detail"> + {c.state} + {running && c.version ? ` · ${c.version}` : ""} + {c.memBytes !== undefined ? ` · ${mem(c.memBytes)}` : ""} + </div> + {pull && pull.active && ( + <div className="po-pull-line"> + {t("Pulling the image")} · {pull.line} + {pull.error ? <span className="text-danger"> — {pull.error}</span> : null} + </div> + )} + </div> + + {/* Enabling it: with the memory warning before a first start (plan 2.4/5.1: ~4 GB RAM recommended at runtime). */} + {!state.enabled && ( + <portal.ui.WarnBanner> + {t("About 4 GB of RAM is recommended for the Document Server while it runs. Its image is pulled on the first start and takes a while; its working data keeps in a container volume, every save comes back to the library.")} + </portal.ui.WarnBanner> + )} + + <portal.ui.SwitchRow + title={t("Use Office")} + detail={t("Starts the Document Server on this machine when switched on, and stops it when switched off. The server keeps it going while it stands (and never brings back one that was stopped on purpose).")} + on={state.enabled} + disabled={busy !== null} + onChange={(on) => void act("settings", () => api.saveSettings({ enabled: on }))} + /> + <div className="flex flex-wrap items-center gap-2"> + <button type="button" className={portal.ui.classes.button} disabled={busy !== null || running} onClick={() => void start()}> + {busy === "start" ? t("Starting… (first pull can take minutes)") : t("Start container")} + </button> + <button type="button" className={portal.ui.classes.button} disabled={busy !== null || !running} onClick={() => void stop()}> + {busy === "stop" ? t("Stopping…") : t("Stop container")} + </button> + </div> + + <portal.ui.Field label={t("Image tag (pinned)")} hint={t("Never pin :latest — a moved image changes the Document Server under you and breaks the co-editing keys. A new tag applies at the next start.")}> + <div className="flex flex-wrap items-center gap-2"> + <input + className={`${portal.ui.classes.input} mt-1 font-mono text-xs`} + style={{ maxWidth: "24rem" }} + value={tag ?? state.imageTag} + aria-label={t("Image tag (pinned)")} + disabled={busy !== null || running} + onChange={(e) => setTag(e.target.value)} + /> + <button type="button" className={portal.ui.classes.primarySmall} disabled={busy !== null || running} onClick={() => void saveTag()}> + {t("Save")} + </button> + </div> + </portal.ui.Field> + + {problem && ( + <portal.ui.ErrorBanner onClose={() => setProblem(null)}>{problem}</portal.ui.ErrorBanner> + )} + + {/* Where the things live, and who can reach them: said once, as fact. */} + <div className="space-y-1"> + <p className="po-exposure">{t("Made documents keep in this extension's own data directory (its docs folder with its index and trash), never in a chat's folder; workspace files stay where they are -- the library indexes them in place, and a download or export is a copy.")}</p> + <p className="po-exposure">{t("The editor itself loads from the Document Server's own origin on this machine, so it works from here over plain HTTP; remote viewers reach the library, but not the live editor (a portal-side proxy would unlock that). The server-facing document fetch and callback endpoints are bound to 127.0.0.1 only.")}</p> + </div> + </div> + ); +} diff --git a/extensions/office/src/web/sort.tsx b/extensions/office/src/web/sort.tsx new file mode 100644 index 000000000..a24cab066 --- /dev/null +++ b/extensions/office/src/web/sort.tsx @@ -0,0 +1,89 @@ +import { LuArrowDownUp } from "react-icons/lu"; +import { portal, t } from "../portal"; +import type { OfficeDoc } from "./api"; + +/** How the office documents sort: one of five keys, one direction. Persisted in localStorage under "po-sort" so + * the sidebar panel and the full Office page share ONE choice per browser; default modified-descending -- newest + * work on top is what a document list should lead with. The sort is pure client-side over the polled docs payload: + * the server hands back the raw meta, ordering is the caller's. */ +export type SortKey = "name" | "modified" | "size" | "type" | "workspace"; +export type SortChoice = { key: SortKey; dir: 1 | -1 }; +const SORT_STORE_KEY = "po-sort"; +export function loadSort(): SortChoice { + try { + const raw = JSON.parse(localStorage.getItem(SORT_STORE_KEY) ?? ""); + if (raw && ["name", "modified", "size", "type", "workspace"].includes(raw.key) && (raw.dir === 1 || raw.dir === -1)) return raw; + } catch { + // never held: fall through to the default + } + return { key: "modified", dir: -1 }; +} +export function saveSort(s: SortChoice): void { + try { + localStorage.setItem(SORT_STORE_KEY, JSON.stringify(s)); + } catch { + // private mode: the choice simply does not stick + } +} + +/** The label the UI shows for a doc's workspace: its scan stamp (the first path segment under its scan root), or + * "library" for docs made in the office library itself -- they have no scan root, so the stamp stays undefined. */ +export const workspaceLabel = (doc: OfficeDoc): string => doc.workspace ?? t("library"); + +/** The one comparator for every key and both call sites (page + panel). `dir` is applied INSIDE, so rules that + * must not flip with direction -- "workspace" keeps its undefineds LAST in either order -- stay independent of it: + * name by locale (the same order listDocs uses), modified by mtimeMs (the file's own revision clock), size in + * bytes, type word-before-cell, workspace by its label. */ +export function compareDocs(a: OfficeDoc, b: OfficeDoc, key: SortKey, dir: 1 | -1 = 1): number { + switch (key) { + case "name": + return a.title.localeCompare(b.title) * dir; + case "modified": + return (a.mtimeMs - b.mtimeMs) * dir; + case "size": + return (a.sizeBytes - b.sizeBytes) * dir; + case "type": + return (a.ext === b.ext ? 0 : a.ext < b.ext ? -1 : 1) * dir; + case "workspace": { + const wa = a.workspace; + const wb = b.workspace; + if (wa === undefined && wb === undefined) return 0; + if (wa === undefined) return 1; // never flipped by dir: library-made docs stay at the bottom + if (wb === undefined) return -1; + return wa.localeCompare(wb) * dir; + } + } +} + +/** The list's sort controls: a key (modified / name / size / type / workspace) and the direction, persisted per + * browser -- compact enough for the sidebar panel, where it sits beside the full page's bar. */ +export function SortBar({ sort, onSort }: { sort: SortChoice; onSort: (next: SortChoice) => void }) { + portal.locale.use(); + return ( + <span className="po-sortbar"> + <portal.ui.Select + size="sm" + value={sort.key} + onChange={(key) => onSort({ ...sort, key })} + aria-label={t("Sort the document list by")} + options={[ + { value: "modified", label: t("Modified") }, + { value: "name", label: t("Name") }, + { value: "size", label: t("Size") }, + { value: "type", label: t("Type") }, + { value: "workspace", label: t("Workspace") }, + ]} + /> + <button + type="button" + className={portal.ui.classes.ghost} + aria-label={sort.dir === -1 ? t("Sorted descending; switch to ascending") : t("Sorted ascending; switch to descending")} + title={sort.dir === -1 ? t("Descending (newest first)") : t("Ascending")} + onClick={() => onSort({ ...sort, dir: sort.dir === -1 ? 1 : -1 })} + > + <LuArrowDownUp aria-hidden /> + {sort.dir === -1 ? t("↓ newest first") : t("↑ oldest first")} + </button> + </span> + ); +} diff --git a/extensions/office/src/web/state.ts b/extensions/office/src/web/state.ts new file mode 100644 index 000000000..0d5ea5b03 --- /dev/null +++ b/extensions/office/src/web/state.ts @@ -0,0 +1,51 @@ +import { useEffect, useSyncExternalStore } from "react"; +import { api, type OfficeState } from "./api"; + +/** + * How the office stands, as the server half last said -- shared by the library page, the per-home chat panels, + * the tool cards and the editor badges. One question at a time, one poll while any of them is on screen: + * GET /state every 5 s (plan 5.7 tier 1), asked again straight after whatever changed it. + */ + +let known: OfficeState | null = null; +const listeners = new Set<() => void>(); +let watchers = 0; +let timer: ReturnType<typeof setInterval> | undefined; +let inflight: Promise<void> | null = null; + +function publish(state: OfficeState): void { + known = state; + for (const listener of listeners) listener(); +} + +/** Ask again now (after a create/rename/delete/settings change), and only once at a time. */ +export function refreshOfficeState(): Promise<void> { + inflight ??= api.state().then(publish, () => {}).finally(() => void (inflight = null)); + return inflight; +} + +const subscribe = (listener: () => void): (() => void) => { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +}; + +/** A hook: the newest answer of GET /state. Starts the shared 5 s poll while a first user mounts, ends it when the last goes. */ +export function useOfficeState(): OfficeState | null { + useEffect(() => { + watchers += 1; + if (watchers === 1) { + void refreshOfficeState(); + timer = setInterval(() => void refreshOfficeState(), 5000); + } + return () => { + watchers -= 1; + if (watchers === 0 && timer !== undefined) { + clearInterval(timer); + timer = undefined; + } + }; + }, []); + return useSyncExternalStore(subscribe, () => known); +} diff --git a/extensions/office/style.css b/extensions/office/style.css new file mode 100644 index 000000000..18d804cf8 --- /dev/null +++ b/extensions/office/style.css @@ -0,0 +1,333 @@ +/* pithagoras-office -- its own chrome. Deliberately NOT the portal card look: an office suite reads as a + workbench -- a thin bar, a list of files like a file manager, chips that say who is in what. The colours are + the portal's theme variables, so a chosen theme paints it too (the same tokens its Tailwind utilities use). */ + +/* ---- the page ---------------------------------------------------------- */ + +.po-page { + max-width: 60rem; + margin: 0 auto; + padding: 1.25rem 1.5rem 3rem; +} + +/* The editor view is the whole page, like a word processor's window: a thin bar and the sheet filling the rest. */ +.po-page.po-editor-page { + display: flex; + flex-direction: column; + height: 100%; + min-height: 0; + max-width: none; + margin: 0; + padding: 0; +} + +/* ---- the editor -------------------------------------------------------- */ + +.po-editor { + display: flex; + flex-direction: column; + flex: 1; + min-height: 0; +} + +.po-editor-bar { + display: flex; + align-items: center; + gap: 0.75rem; + flex-wrap: wrap; + padding: 0.4rem 0.75rem; + border-bottom: 1px solid rgb(var(--line)); + background: rgb(var(--surface)); +} + +.po-editor-title { + font-weight: 600; + font-size: 0.9rem; + color: rgb(var(--fg)); + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.po-editor-spacer { + flex: 1; +} + +.po-host { + position: relative; + flex: 1; + min-height: 0; + background: rgb(var(--canvas)); +} + +/* The Document Server fills its own iframe into the placeholder; make sure it takes the whole place. */ +.po-host iframe { + position: absolute; + inset: 0; + width: 100%; + height: 100%; + border: 0; +} + +.po-loading { + position: absolute; + inset: 0; + display: flex; + align-items: center; + justify-content: center; + font-size: 0.85rem; + color: rgb(var(--fg-muted)); +} + +/* ---- the document list -------------------------------------------------- */ + +.po-listhead { + display: flex; + align-items: center; + justify-content: space-between; + gap: 0.5rem; + flex-wrap: wrap; +} + +.po-sortbar, +.po-rescan { + display: inline-flex; + align-items: center; + gap: 0.5rem; +} + +.po-list { + display: flex; + flex-direction: column; + gap: 0.4rem; +} + +.po-row { + display: flex; + align-items: center; + gap: 0.75rem; + padding: 0.55rem 0.75rem; + border: 1px solid rgb(var(--line)); + border-radius: 0.6rem; + background: rgb(var(--surface)); +} + +.po-row:hover { + border-color: rgb(var(--fg-subtle)); + background: rgb(var(--raised) / 0.5); +} + +/* Where there is nothing to point with, the row's buttons are always there. */ +@media (hover: none) { + .po-row-actions { visibility: visible; } +} + +.po-open { + flex: 1; + min-width: 0; + display: flex; + align-items: center; + gap: 0.6rem; + border: 0; + background: none; + padding: 0; + text-align: left; + cursor: pointer; +} + +.po-title { + font-size: 0.9rem; + font-weight: 500; + color: rgb(var(--fg)); + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.po-meta { + margin-top: 0.15rem; + font-size: 11px; + color: rgb(var(--fg-subtle)); + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.po-row-actions { + display: flex; + align-items: center; + gap: 0.25rem; + visibility: hidden; +} + +.po-row:hover .po-row-actions, +.po-row:focus-within .po-row-actions { + visibility: visible; +} + +/* Format marks: a word document and a sheet are told apart at a glance, the way an office's file lists do it. */ +.po-ext { + flex: none; + width: 18px; + height: 18px; +} +.po-ext-word { color: rgb(var(--accent)); } +.po-ext-cell { color: rgb(var(--ok)); } + +/* ---- chips: who owns it, who is in it ----------------------------------- */ + +.po-chips { + display: flex; + align-items: center; + gap: 0.3rem; + flex-wrap: wrap; +} + +.po-chip { + display: inline-flex; + align-items: center; + gap: 0.25rem; + padding: 0.1rem 0.5rem; + border: 1px solid rgb(var(--line)); + border-radius: 999px; + font-size: 11px; + line-height: 1.4; + color: rgb(var(--fg-muted)); + background: rgb(var(--raised) / 0.6); + max-width: 14rem; +} + +.po-chip svg { flex: none; } + +.po-chip-owner { + border-color: rgb(var(--accent) / 0.5); + color: rgb(var(--accent)); +} + +.po-chip-lock { + border-color: rgb(var(--warn) / 0.6); + background: rgb(var(--warn) / 0.12); + color: rgb(var(--fg)); +} + +/* "The agent changed it while you had it open" -- the tier-1 badge of plan 5.7, until the refresh handshake lands. */ +.po-chip-stale { + border-color: rgb(var(--danger) / 0.6); + background: rgb(var(--danger) / 0.12); + color: rgb(var(--fg)); +} + +.po-refresh { + border: 0; + background: none; + padding: 0; + cursor: pointer; + font-size: 11px; + color: inherit; + text-decoration: underline; +} + +/* ---- forms on the page --------------------------------------------------- */ + +.po-form { + display: flex; + align-items: center; + gap: 0.5rem; + flex-wrap: wrap; + padding: 0.6rem 0.75rem; + border: 1px dashed rgb(var(--line)); + border-radius: 0.6rem; +} + +.po-form input { min-width: 0; } +.po-form .po-grow { flex: 1; min-width: 8rem; } + +.po-hint { + font-size: 11px; + color: rgb(var(--fg-subtle)); +} + +/* ---- the sidebar panel's slim new-document row ---------------------------- + The full page keeps its NewDocument form; beside a chat the same create flow (no owner: unowned docs are + shared) goes in one slim dashed row with an icon kind toggle, so the panel stays list-shaped. */ + +.po-newcompact { + display: flex; + align-items: center; + gap: 0.4rem; + flex-wrap: wrap; + padding: 0.35rem 0.5rem; + border: 1px dashed rgb(var(--line)); + border-radius: 0.6rem; +} + +.po-newcompact input { + min-width: 0; + flex: 1; +} + +.po-kind { + display: inline-flex; + align-items: center; + gap: 0.25rem; +} + +.po-kind-btn { + display: inline-flex; + align-items: center; + padding: 0.15rem 0.4rem; + border: 1px solid rgb(var(--line)); + border-radius: 0.5rem; + background: none; + cursor: pointer; + color: rgb(var(--fg-muted)); +} + +.po-kind-on { + border-color: rgb(var(--accent) / 0.6); + background: rgb(var(--accent) / 0.08); + color: rgb(var(--accent)); +} + +/* ---- the settings page's service panel ----------------------------------- */ + +.po-service { + display: flex; + flex-direction: column; + gap: 0.35rem; + padding: 0.75rem 1rem; + border: 1px solid rgb(var(--line)); + border-left: 3px solid rgb(var(--accent)); + border-radius: 0.6rem; + background: rgb(var(--raised) / 0.4); +} + +.po-service-head { + display: flex; + align-items: center; + gap: 0.5rem; + font-size: 0.9rem; + font-weight: 600; + color: rgb(var(--fg)); +} + +.po-state-idle { color: rgb(var(--fg-subtle)); } +.po-state-live { color: rgb(var(--ok)); } +.po-state-pulling { color: rgb(var(--warn)); } + +.po-service-detail { + font-size: 12px; + color: rgb(var(--fg-muted)); +} + +.po-pull-line { + font-family: ui-monospace, monospace; + font-size: 11px; + color: rgb(var(--fg-muted)); + overflow-wrap: anywhere; +} + +.po-exposure { + font-size: 12px; + color: rgb(var(--fg-subtle)); +} diff --git a/extensions/office/test/callback.test.mts b/extensions/office/test/callback.test.mts new file mode 100644 index 000000000..529bcc2f1 --- /dev/null +++ b/extensions/office/test/callback.test.mts @@ -0,0 +1,598 @@ +/** + * Unit tests for src/callback.ts (+ src/live.ts): the DS callback state machine of plan 5.3. Everything is + * self-contained -- tmp data dirs, self-minted HS256 JWTs, a LOCAL node:http server standing in for the + * Document Server's saved-file/changes endpoints, and synthetic IncomingMessage/ServerResponse pairs (a + * Readable stream + a recording object). Nothing here reaches portal() or the real data dir: every handler is + * built with createCallbackHandler with an explicit root; only the bare handleCallback/configure test touches + * module-level wiring (and still only status-4, which writes nothing). + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import http from "node:http"; +import { Readable } from "node:stream"; +import { createHmac, createHash } from "node:crypto"; +import type { IncomingMessage, ServerResponse } from "node:http"; +import { signJwt, verifyJwt } from "../src/jwt"; +import { + CallbackState, configure, createCallbackHandler, handleCallback, makeKey, onPersisted, parseKey, revHashOf, +} from "../src/callback"; +import { createDoc, getDoc } from "../src/store"; +import * as live from "../src/live"; + +const SECRET = "unit-test-secret-0123456789abcdef"; +const DS_STATE: CallbackState = { containerId: "cid-1", origin: "http://127.0.0.1", imageTag: "onlyoffice/documentserver:10.0" }; + +function tmpRoot(): string { + return fs.mkdtempSync(path.join(os.tmpdir(), "office-callback-")); +} + +const docFile = (root: string, meta: { docId: string; title: string; ext: string }): string => + path.join(root, "docs", meta.docId, `${meta.title}.${meta.ext}`); + +/* ------------------------------------------------------------------ */ +/* synthetic req/res + the local "DS" file server */ +/* ------------------------------------------------------------------ */ + +type FakeRes = { + statusCode: number; + body: string; + headersSent: boolean; + ended: boolean; + finished: Promise<void>; + writeHead(code: number, headers?: Record<string, string>): void; + end(chunk?: string | Buffer): void; +}; + +function makeRes(): FakeRes { + const res = { statusCode: 0, body: "", headersSent: false, ended: false, _done: null as null | (() => void) } as FakeRes & { _done: null | (() => void) }; + res.finished = new Promise<void>((r) => (res._done = r)); + res.writeHead = function (code: number): void { + this.statusCode = code; + this.headersSent = true; + }; + res.end = function (chunk?: string | Buffer): void { + if (chunk !== undefined && chunk !== null) this.body += chunk.toString(); + this.ended = true; + res._done?.(); + }; + return res; +} + +function makeReq(method: string, url: string, headers: Record<string, string>, bodyObj: unknown): IncomingMessage { + const stream = Readable.from([Buffer.from(JSON.stringify(bodyObj))]); + Object.assign(stream, { method, url, headers }); + return stream as unknown as IncomingMessage; +} + +type Handler = (req: IncomingMessage, res: ServerResponse) => void; + +async function post(handler: Handler, body: Record<string, unknown>, opts: { method?: string; url?: string; headers?: Record<string, string> } = {}): Promise<FakeRes> { + const res = makeRes(); + handler(makeReq(opts.method ?? "POST", opts.url ?? "/callback", opts.headers ?? {}, body), res as unknown as ServerResponse); + await res.finished; + return res; +} + +const json = (res: FakeRes): Record<string, unknown> => JSON.parse(res.body) as Record<string, unknown>; + +/** A local node:http server standing in for the DS saved-file/changes endpoints. onSaved(n) decides the n-th + * GET /saved outcome; changes is served verbatim at /changes.zip. Records every X-Auth-Token it was sent. */ +type FileServer = { + base: string; + savedAttempts: number; + tokens: string[]; + close(): Promise<void>; +}; + +async function startFileServer( + onSaved: (n: number) => { status: number; bytes: Buffer }, + changes: Buffer | null, +): Promise<FileServer> { + const server = http.createServer((req, res) => { + let n = 0; + if (req.url === "/saved") { + n = ++serverSavedCalls; + serverTokens.push(typeof req.headers["x-auth-token"] === "string" ? req.headers["x-auth-token"] : ""); + const out = onSaved(n); + res.writeHead(out.status, { "Content-Type": "application/octet-stream" }); + res.end(out.bytes); + } else if (req.url === "/changes.zip") { + if (!changes) return res.writeHead(404).end(); + serverTokens.push(typeof req.headers["x-auth-token"] === "string" ? req.headers["x-auth-token"] : ""); + res.writeHead(200, { "Content-Type": "application/zip" }); + res.end(changes); + } else { + res.writeHead(404); + res.end(); + } + }); + let serverSavedCalls = 0; + const serverTokens: string[] = []; + await new Promise<void>((r) => server.listen(0, "127.0.0.1", r)); + const addr = server.address() as { port: number }; + return { + base: `http://127.0.0.1:${addr.port}`, + get savedAttempts() { + return serverSavedCalls; + }, + get tokens() { + return serverTokens; + }, + close: () => + new Promise<void>((r, j) => { + // undici's fetch pool keep-ALIVEs its sockets; force them down so close() resolves immediately + server.closeAllConnections?.(); + server.close((e) => (e ? j(e) : r())); + }), + }; +} + +/* ------------------------------------------------------------------ */ +/* parseKey */ +/* ------------------------------------------------------------------ */ + +test("parseKey: valid round-trip (sha256 of '12345', base64url no pad)", () => { + const sha = createHash("sha256").update("12345").digest("base64url"); + assert.equal(sha.length, 43); // one unpadded sha256 digest in base64url + assert.equal(revHashOf(12345), sha); + assert.equal(makeKey("my-doc", 12345), `office|my-doc|${sha}`); + const parsed = parseKey(`office|my-doc|${sha}`); + assert.deepEqual(parsed, { docId: "my-doc", revHash: sha }); +}); + +test("parseKey: a docId containing a pipe still parses (greedy first/last split)", () => { + const key = makeKey("a|b", 42); + assert.deepEqual(parseKey(key), { docId: "a|b", revHash: revHashOf(42) }); +}); + +test("parseKey: null on missing parts, empty docId and malformed hash part", () => { + const sha = revHashOf(1); + assert.equal(parseKey(""), null); + assert.equal(parseKey(`my-doc|${sha}`), null); // no "office|" prefix + assert.equal(parseKey("office|my-doc"), null); // missing hash part + assert.equal(parseKey(`office|my-doc|`), null); // empty hash part + assert.equal(parseKey(`office||${sha}`), null); // empty docId + assert.equal(parseKey(`office|doc|x`.padEnd(7 + 4 + 50, "z")), null); // wrong length / alphabet + assert.equal(parseKey("office|doc|" + sha.slice(0, 42)), null); // 42 chars -- not a sha256 digest + assert.equal(parseKey("office|doc|" + "=".repeat(43)), null); // "=" is not base64url +}); + +/* ------------------------------------------------------------------ */ +/* status-2 happy path */ +/* ------------------------------------------------------------------ */ + +const hookLog: { docId: string; buffer: Buffer }[] = []; +onPersisted((docId, buffer) => hookLog.push({ docId, buffer })); // single module-level collector, cleared per test + +test("status 2 happy path: persists bytes, bumps index, stores history zip, fires onPersisted, unlocks", async () => { + const root = tmpRoot(); + const sink: string[] = []; + const doc = await createDoc({ title: "Quarterly Report", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + hookLog.length = 0; + + live.setUser(doc.docId, { id: "u1", name: "Ana" }, true); // the human editor is open -> locked via presence + assert.equal(live.pollState(doc.docId, 0).locked, true); + + const newBytes = Buffer.from("saved-by-ds-content-" + "x".repeat(257)); + const zipBytes = Buffer.from("CHANGES-ARCHIVE-BYTES"); + const server = await startFileServer(() => ({ status: 200, bytes: newBytes }), zipBytes); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, retryDelaysMs: [5, 10], log: (l) => sink.push(l) }); + + // The envelope is bound to the signed JWT claims; the JSON body is ignored by the handler. + const envelope = { + key: makeKey(doc.docId, before.mtimeMs), + status: 2, + url: `${server.base}/saved`, + changesurl: `${server.base}/changes.zip`, + }; + const res = await post(handler, envelope, { headers: { "x-auth-token": signJwt(envelope, 60_000, SECRET) } }); + + await server.close(); + assert.equal(res.statusCode, 200); + assert.deepEqual(json(res), { error: 0 }); + + // the file bytes changed to exactly what DS served + assert.ok(fs.readFileSync(docFile(root, doc)).equals(newBytes), "document bytes were replaced"); + // the index moved on: new mtime revision + size of the new bytes + const after = getDoc(doc.docId, root)!; + assert.notEqual(after.mtimeMs, before.mtimeMs); + assert.equal(after.sizeBytes, newBytes.length); + // the changes archive landed at history/<doc>/<newRev>.zip (rev = the NEW post-persist mtime) + const zipPath = path.join(root, "history", doc.docId, `${after.mtimeMs}.zip`); + assert.ok(fs.readFileSync(zipPath).equals(zipBytes), "changes zip stored under the new revision"); + // onPersisted fired once for this doc with the exact buffer (ping's wake hook will see these bytes) + const mine = hookLog.filter((h) => h.docId === doc.docId); + assert.equal(mine.length, 1); + assert.ok(mine[0].buffer.equals(newBytes)); + // the persist path did not need retries and authenticated its outbound GETs with a verifiable HS256 token + assert.equal(server.savedAttempts, 1); + const t = server.tokens.find((x) => x !== ""); + assert.ok(t && verifyJwt(t, SECRET), "outbound fetch carried a valid X-Auth-Token JWT"); + // lock released after the persist (status-2 row: unlock AFTER persist) + assert.equal(live.pollState(doc.docId, 0).locked, false); +}); + +test("stale key (revHash of an older mtime): document NOT overwritten, JSON error, one log line", async () => { + const root = tmpRoot(); + const sink: string[] = []; + const doc = await createDoc({ title: "Stale Doc", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + const original = fs.readFileSync(docFile(root, doc)); + hookLog.length = 0; + + const server = await startFileServer(() => ({ status: 200, bytes: Buffer.from("should-never-land") }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, retryDelaysMs: [5, 10], log: (l) => sink.push(l) }); + + // key minted from an OLDER mtime than the one currently in the index + const envelope = { + key: makeKey(doc.docId, before.mtimeMs - 60_000), + status: 2, + url: `${server.base}/saved`, + }; + const res = await post(handler, envelope, { headers: { "x-auth-token": signJwt(envelope, 60_000, SECRET) } }); + + await server.close(); + assert.equal(res.statusCode, 200); + assert.deepEqual(json(res), { error: 1 }); // non-zero -> the editor shows a save failure, per plan 4 + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original), "file untouched"); + const after = getDoc(doc.docId, root)!; + assert.equal(after.mtimeMs, before.mtimeMs); + assert.equal(after.sizeBytes, original.length); + assert.equal(server.savedAttempts, 0, "stale gate refuses BEFORE the download"); + assert.equal(hookLog.filter((h) => h.docId === doc.docId).length, 0); + assert.ok(!fs.existsSync(path.join(root, "history", doc.docId))); + assert.ok(sink.some((l) => l.includes("stale key") && l.includes(doc.docId)), `a stale-key line was logged: ${sink.join("; ")}`); +}); + +test("duplicate force-save pass (same url + our revision): treated as done, no second write", async () => { + const root = tmpRoot(); + const sink: string[] = []; + const doc = await createDoc({ title: "Forced Doc", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + hookLog.length = 0; + + const firstBytes = Buffer.from("first-forcesave"); + let n = 0; + const server = await startFileServer(() => ({ status: 200, bytes: firstBytes }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, retryDelaysMs: [5, 10], log: (l) => sink.push(l) }); + // Pass 1: the status-1 forcesave action (actions[].type === 2) persists now, doc STAYS locked. + live.setLocked(doc.docId, true); + const env1 = { + key: makeKey(doc.docId, before.mtimeMs), + status: 1, + users: [{ id: "u2", name: "Bob" }], + actions: [{ type: 2, userid: "u2" }], + url: `${server.base}/saved`, + }; + const r1 = await post(handler, env1, { headers: { "x-auth-token": signJwt(env1, 60_000, SECRET) } }); + assert.deepEqual(json(r1), { error: 0 }); + const mid = getDoc(doc.docId, root)!; + assert.ok(fs.readFileSync(docFile(root, doc)).equals(firstBytes)); + assert.equal(live.pollState(doc.docId, 0).locked, true, "forcesave keeps the doc locked (status-6 rules)"); + + // Pass 2: DS then reports the canonical status 6 for the SAME save. The key now hashes an older mtime than + // the one we just wrote -- without lastPersist this would answer {error:1} as a stale session. + const env2 = { + key: makeKey(doc.docId, before.mtimeMs), + status: 6, + url: `${server.base}/saved`, + forcesavetype: 1, + }; + const r2 = await post(handler, env2, { headers: { "x-auth-token": signJwt(env2, 60_000, SECRET) } }); + assert.deepEqual(json(r2), { error: 0 }, "duplicate pass reports success"); + const after = getDoc(doc.docId, root)!; + assert.equal(after.mtimeMs, mid.mtimeMs, "no second persist happened"); + assert.equal(hookLog.filter((h) => h.docId === doc.docId).length, 1, "hook fired once for the single real save"); + await server.close(); +}); + +/* ------------------------------------------------------------------ */ +/* JWT verification */ +/* ------------------------------------------------------------------ */ + +test("JWT: garbage token -> 401 {error:1} without touching disk", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Jwt Doc", type: "word" }, root); + const original = fs.readFileSync(docFile(root, doc)); + const server = await startFileServer(() => ({ status: 200, bytes: Buffer.from("nope") }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + + const res = await post(handler, { + key: makeKey(doc.docId, getDoc(doc.docId, root)!.mtimeMs), + status: 2, + url: `${server.base}/saved`, + }, { headers: { "x-auth-token": "garbage.token.value" } }); + + await server.close(); + assert.equal(res.statusCode, 401); + assert.deepEqual(json(res), { error: 1 }); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original), "no download, no write"); + assert.equal(server.savedAttempts, 0); +}); + +test("JWT: RS256-alg token with a valid HMAC signature is still rejected (alg pinned to HS256)", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Alg Doc", type: "word" }, root); + const original = fs.readFileSync(docFile(root, doc)); + + // Forge a token whose header says RS256 but whose signature is the HS256 MAC over that exact pair. + const b64url = (s: string | Buffer) => Buffer.from(s).toString("base64url"); + const head = b64url(JSON.stringify({ alg: "RS256", typ: "JWT" })); + const body = b64url(JSON.stringify({ key: makeKey(doc.docId, getDoc(doc.docId, root)!.mtimeMs), status: 2 })); + const sig = b64url(createHmac("sha256", SECRET).update(`${head}.${body}`).digest()); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + + const res = await post(handler, { status: 2 }, { headers: { "x-auth-token": `${head}.${body}.${sig}` } }); + assert.equal(res.statusCode, 401); + assert.deepEqual(json(res), { error: 1 }); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original)); +}); + +test("JWT: expired token -> 401; missing token entirely -> 401", async () => { + const root = tmpRoot(); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + const key = makeKey("whatever-doc", 1); + + const expired = await post(handler, { key, status: 4 }, { headers: { "x-auth-token": signJwt({ sub: "x" }, -1000, SECRET) } }); + assert.equal(expired.statusCode, 401); + assert.deepEqual(json(expired), { error: 1 }); + + const missing = await post(handler, { key, status: 4 }); // no x-auth-token header at all + assert.equal(missing.statusCode, 401); + assert.deepEqual(json(missing), { error: 1 }); +}); + +test("JWT: embedded body.token is accepted as a defensive fallback (header path stays canonical)", async () => { + const root = tmpRoot(); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + live.setLocked("fallback-doc", true); + + const res = await post(handler, { + key: makeKey("fallback-doc", 1), + status: 4, + // The fallback token itself must carry the signed envelope (the body is never trusted). + token: signJwt({ key: makeKey("fallback-doc", 1), status: 4 }, 60_000, SECRET), // no x-auth-token header + }); + assert.equal(res.statusCode, 200); + assert.deepEqual(json(res), { error: 0 }); + assert.equal(live.pollState("fallback-doc", 0).locked, false, "status-4 unlock applied through the fallback"); +}); + +test("envelope binding: valid token without signed key/status cannot drive a hostile body", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Bind Doc", type: "word" }, root); + const original = fs.readFileSync(docFile(root, doc)); + const server = await startFileServer(() => ({ status: 200, bytes: Buffer.from("attacker-bytes") }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + + // A valid token that carries NO envelope (the shape of our own non-callback tokens) plus a hostile JSON + // body claiming status 2 for a library document: rejected at the binding check, disk untouched. + const res = await post(handler, { + key: makeKey(doc.docId, getDoc(doc.docId, root)!.mtimeMs), + status: 2, + url: `${server.base}/saved`, + }, { headers: { "x-auth-token": signJwt({ sub: "x" }, 60_000, SECRET) } }); + + await server.close(); + assert.equal(res.statusCode, 400); + assert.deepEqual(json(res), { error: 1 }); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original), "hostile body ignored, file untouched"); + assert.equal(server.savedAttempts, 0); +}); + +/* ------------------------------------------------------------------ */ +/* status 1 / 4 presence + lock commands */ +/* ------------------------------------------------------------------ */ + +test("status 1 join then leave mutates live.usersOf; status 4 unlocks without any disk write", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Presence Doc", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + const original = fs.readFileSync(docFile(root, doc)); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + // JOIN: presence lands in live with id/name and a numeric join time; the doc locks (users.length>0). + const envJoin = { key: makeKey(doc.docId, before.mtimeMs), status: 1, users: [{ id: "u9", name: "Bob" }] }; + const join = await post(handler, envJoin, { headers: { "x-auth-token": signJwt(envJoin, 60_000, SECRET) } }); + assert.deepEqual(json(join), { error: 0 }); + const joined = live.usersOf(doc.docId); + assert.equal(joined.length, 1); + assert.equal(joined[0].id, "u9"); + assert.equal(joined[0].name, "Bob"); + assert.equal(typeof joined[0].at, "number"); + assert.equal(live.pollState(doc.docId, 0).locked, true); + + // LEAVE: the empty users snapshot removes the user and clears the derived lock. + const envLeave = { key: makeKey(doc.docId, before.mtimeMs), status: 1, users: [] }; + const leave = await post(handler, envLeave, { headers: { "x-auth-token": signJwt(envLeave, 60_000, SECRET) } }); + assert.deepEqual(json(leave), { error: 0 }); + assert.deepEqual(live.usersOf(doc.docId), []); + assert.equal(live.pollState(doc.docId, 0).locked, false); + + // STATUS 4: explicit unlock, and the file + index are byte-identical to before. + live.setLocked(doc.docId, true); + const env4 = { key: makeKey(doc.docId, before.mtimeMs), status: 4 }; + const r4 = await post(handler, env4, { headers: { "x-auth-token": signJwt(env4, 60_000, SECRET) } }); + assert.deepEqual(json(r4), { error: 0 }); + assert.equal(live.pollState(doc.docId, 0).locked, false); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original), "status 4 wrote nothing"); + const after = getDoc(doc.docId, root)!; + assert.equal(after.mtimeMs, before.mtimeMs); + assert.equal(after.sizeBytes, before.sizeBytes); +}); + +test("statuses 3 and 7: one log line each, zero file changes, zero downloads", async () => { + const root = tmpRoot(); + const sink: string[] = []; + const doc = await createDoc({ title: "Errors Doc", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + const original = fs.readFileSync(docFile(root, doc)); + + const server = await startFileServer(() => ({ status: 200, bytes: Buffer.from("should-not-be-fetched") }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, retryDelaysMs: [5, 10], log: (l) => sink.push(l) }); + const env3 = { key: makeKey(doc.docId, before.mtimeMs), status: 3, url: `${server.base}/saved` }; + const r3 = await post(handler, env3, { headers: { "x-auth-token": signJwt(env3, 60_000, SECRET) } }); + assert.equal(r3.statusCode, 200); + assert.deepEqual(json(r3), { error: 0 }, "we did not fail internally, so the answer is 0"); + + const env7 = { key: makeKey(doc.docId, before.mtimeMs), status: 7, url: `${server.base}/saved`, forcesavetype: 2 }; + const r7 = await post(handler, env7, { headers: { "x-auth-token": signJwt(env7, 60_000, SECRET) } }); + assert.equal(r7.statusCode, 200); + assert.deepEqual(json(r7), { error: 0 }); + + await server.close(); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original)); + const after = getDoc(doc.docId, root)!; + assert.equal(after.mtimeMs, before.mtimeMs); + assert.equal(server.savedAttempts, 0, "no fetch happened for 3/7"); + assert.ok(sink.some((l) => l.startsWith("status-3:")), `status-3 line present: ${sink.join("; ")}`); + assert.ok(sink.some((l) => l.startsWith("status-7:") && l.includes("forcesavetype=2")), `status-7 line present`); +}); + +/* ------------------------------------------------------------------ */ +/* retries */ +/* ------------------------------------------------------------------ */ + +test("retry: two failed downloads then success -> persisted", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Flaky Doc", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + hookLog.length = 0; + + const saved = Buffer.from("eventually-arrives"); + const server = await startFileServer((n) => (n < 3 ? { status: 500, bytes: Buffer.alloc(0) } : { status: 200, bytes: saved }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, retryDelaysMs: [5, 10], log: () => {} }); + + const envelope = { + key: makeKey(doc.docId, before.mtimeMs), + status: 2, + url: `${server.base}/saved`, + }; + const res = await post(handler, envelope, { headers: { "x-auth-token": signJwt(envelope, 60_000, SECRET) } }); + + await server.close(); + assert.deepEqual(json(res), { error: 0 }); + assert.equal(server.savedAttempts, 3, "failed twice, succeeded on the third attempt"); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(saved)); + assert.equal(hookLog.filter((h) => h.docId === doc.docId).length, 1); +}); + +test("retry: all attempts fail -> JSON error, doc untouched, lock left as-is", async () => { + const root = tmpRoot(); + const sink: string[] = []; + const doc = await createDoc({ title: "Dead Doc", type: "word" }, root); + const before = getDoc(doc.docId, root)!; + const original = fs.readFileSync(docFile(root, doc)); + + const server = await startFileServer(() => ({ status: 503, bytes: Buffer.alloc(0) }), null); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, retryDelaysMs: [5, 10], log: (l) => sink.push(l) }); + + live.setLocked(doc.docId, true); + const envelope = { + key: makeKey(doc.docId, before.mtimeMs), + status: 2, + url: `${server.base}/saved`, + }; + const res = await post(handler, envelope, { headers: { "x-auth-token": signJwt(envelope, 60_000, SECRET) } }); + + await server.close(); + assert.equal(res.statusCode, 200); // DS reads the JSON error field, not the HTTP code (plan 4) + assert.deepEqual(json(res), { error: 1 }); + assert.equal(server.savedAttempts, 3, "1 + retryDelaysMs.length attempts"); + assert.ok(fs.readFileSync(docFile(root, doc)).equals(original), "existing file never deleted or replaced"); + const after = getDoc(doc.docId, root)!; + assert.equal(after.mtimeMs, before.mtimeMs); + assert.equal(live.pollState(doc.docId, 0).locked, true, "fetch failure leaves the lock state untouched (DS will retry)"); + assert.ok(sink.some((l) => l.includes("could not download")), `a download-failure line was logged`); +}); + +/* ------------------------------------------------------------------ */ +/* envelope / route rejections */ +/* ------------------------------------------------------------------ */ + +test("route + envelope: GET -> 405, bad path -> 404, unknown status / bad key -> 400", async () => { + const root = tmpRoot(); + const handler = createCallbackHandler(() => DS_STATE, () => SECRET, { root, log: () => {} }); + const key = makeKey("route-doc", 1); + // Envelopes are signed into the token claims; each post below signs its own envelope. + const tokenFor = (env: Record<string, unknown>) => ({ "x-auth-token": signJwt(env, 60_000, SECRET) }); + + const get = await post(handler, { key, status: 4 }, { method: "GET", headers: tokenFor({ key, status: 4 }) }); + assert.equal(get.statusCode, 405); + assert.deepEqual(json(get), { error: 1 }); + + const wrongPath = await post(handler, { key, status: 4 }, { url: "/doc/something-else", headers: tokenFor({ key, status: 4 }) }); + assert.equal(wrongPath.statusCode, 404); + assert.deepEqual(json(wrongPath), { error: 1 }); + + // Signed envelope with an unknown status value -> the handler rejects the status itself. + const unknownStatus = await post(handler, { key, status: 99 }, { headers: tokenFor({ key, status: 99 }) }); + assert.equal(unknownStatus.statusCode, 400); + assert.deepEqual(json(unknownStatus), { error: 1 }); + + // Signed envelope with a malformed key -> parseKey rejects it. + const badKey = await post(handler, { key: "nope", status: 2 }, { headers: tokenFor({ key: "nope", status: 2 }) }); + assert.equal(badKey.statusCode, 400); + assert.deepEqual(json(badKey), { error: 1 }); + + // No signed key in the claims -> rejected at the binding check (the unsigned body is never consulted). + const noKey = await post(handler, { key, status: 2 }, { headers: tokenFor({ status: 2 }) }); + assert.equal(noKey.statusCode, 400); + assert.deepEqual(json(noKey), { error: 1 }); +}); + +/* ------------------------------------------------------------------ */ +/* the bare module-level handler + configure (T-WIRE wiring) */ +/* ------------------------------------------------------------------ */ + +test("bare handleCallback: 503 until configure; after configure it serves requests", async () => { + const root = tmpRoot(); + + // Not configured yet (no test before this one called configure). + const notYet = await post(handleCallback, { key: makeKey("wire-doc", 1), status: 4 }); + assert.equal(notYet.statusCode, 503); + assert.deepEqual(json(notYet), { error: 1 }); + + // Configured but the DS record is still absent (getState -> null) -> 503 as well. + configure({ getState: () => null, getSecret: () => SECRET, root, log: () => {} }); + const noState = await post(handleCallback, { key: makeKey("wire-doc", 1), status: 4 }); + assert.equal(noState.statusCode, 503); + assert.deepEqual(json(noState), { error: 1 }); + + // The real wiring shape (what server.ts does at boot). + live.setLocked("wire-doc", true); + configure({ getState: () => DS_STATE, getSecret: () => SECRET, root, log: () => {} }); + const wireEnv = { key: makeKey("wire-doc", 1), status: 4 }; + const ok = await post(handleCallback, wireEnv, { headers: { "x-auth-token": signJwt(wireEnv, 60_000, SECRET) } }); + assert.equal(ok.statusCode, 200); + assert.deepEqual(json(ok), { error: 0 }); + assert.equal(live.pollState("wire-doc", 0).locked, false); +}); + +/* ------------------------------------------------------------------ */ +/* live.ts pollState resolver */ +/* ------------------------------------------------------------------ */ + +test("live.pollState: pendingRev only when the resolved revision is strictly newer than the client's", () => { + live.resetLive(); // clean slate (also proves reset exists and wipes state) + const docId = "resolver-doc"; + live.setMetaResolver((id) => (id === docId ? 2000 : null)); + + assert.deepEqual(live.pollState(docId, 1999), { locked: false, pendingRev: 2000 }); // agent updated since open + assert.deepEqual(live.pollState(docId, 2000), { locked: false, pendingRev: null }); // same revision + assert.deepEqual(live.pollState(docId, 5000), { locked: false, pendingRev: null }); // client ahead (weird but safe) + + live.setUser(docId, { id: "u", name: "Z" }, true); + assert.equal(live.pollState(docId, 2000).locked, true, "presence locks even without the flag"); + live.setLocked(docId, false); + live.clearDoc(docId); + assert.deepEqual(live.usersOf(docId), []); + assert.equal(live.pollState(docId, 1999).locked, false, "clearDoc drops the lock entirely"); + + // A cleared resolver makes pendingRev null again (server.ts re-registers at boot). + live.setMetaResolver(null); + assert.deepEqual(live.pollState(docId, 1999), { locked: false, pendingRev: null }); +}); diff --git a/extensions/office/test/container.test.mts b/extensions/office/test/container.test.mts new file mode 100644 index 000000000..94e6af6a8 --- /dev/null +++ b/extensions/office/test/container.test.mts @@ -0,0 +1,343 @@ +/** + * Unit tests for src/container.ts: DS container lifecycle over a MOCK Docker (object literal cast as Docker) that + * records every call, plus real tmp dirs for settings/jwt-secret/ds.json/user-stopped. The mock never reaches the + * socket and nothing here reaches portal() -- docker and root are always passed explicitly. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import type { ContainerInfo, Docker, ExecResult, VolumeInfo } from "../../docker/src/api"; +import { DS_NAME, DS_VOLUME, ensureContainer, lookAfter, readDsRecord, status, stopContainer, userStopped } from "../src/container"; + +const DEFAULT_TAG = "onlyoffice/documentserver:10.0"; +const TAG92 = "onlyoffice/documentserver:9.2"; + +function tmpRoot(withSettings?: Record<string, unknown>, secret = "s3cret\n"): string { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-container-")); + if (withSettings) fs.writeFileSync(path.join(root, "settings.json"), JSON.stringify({ enabled: false, imageTag: DEFAULT_TAG, ...withSettings })); + else fs.writeFileSync(path.join(root, "settings.json"), JSON.stringify({ enabled: false, imageTag: DEFAULT_TAG })); + if (secret !== null) fs.writeFileSync(path.join(root, "jwt-secret"), secret); + return root; +} + +type MockState = { + availableFlag: boolean; + networkMode: string; + container: ContainerInfo | undefined; + volumes: VolumeInfo[]; + inspectBody: Record<string, unknown> | undefined; + execResult: ExecResult; +}; + +function makeDocker(opts: { networkMode?: string } = {}) { + const state: MockState = { + availableFlag: true, + networkMode: opts.networkMode ?? "host", + container: undefined, + volumes: [], + inspectBody: undefined, + execResult: { output: "", exitCode: 1, timedOut: false }, + }; + const calls = { + available: 0, + ensure: [] as string[], + get: [] as string[], + create: [] as Array<[string, Record<string, unknown>]>, + start: [] as string[], + stop: [] as string[], + inspect: [] as string[], + exec: [] as Array<[string, string[]]>, + volumeList: 0, + volumeCreate: [] as string[], + sharedNetwork: 0, + }; + const docker = { + available: async () => { + calls.available++; + return state.availableFlag; + }, + sharedNetwork: async () => { + calls.sharedNetwork++; + return state.networkMode; + }, + images: { + present: async (image: string) => true, + ensure: async (image: string, onProgress?: (s: { active: boolean; line: string; error?: string }) => void) => { + calls.ensure.push(image); + onProgress?.({ active: true, line: "downloading layers…" }); + onProgress?.({ active: false, line: "done" }); + }, + }, + containers: { + list: async () => (state.container ? [state.container] : []), + get: async (name: string) => { + calls.get.push(name); + return state.container; + }, + create: async (name: string, spec: Record<string, unknown>) => { + calls.create.push([name, spec]); + const info: ContainerInfo = { name, id: `cid-${calls.create.length}`, image: String(spec.Image), state: "created", status: "Created", owner: "", labels: {} }; + state.container = info; + return info; + }, + start: async (name: string) => { + calls.start.push(name); + if (state.container && name === DS_NAME) state.container.state = "running"; + }, + stop: async (name: string) => { + calls.stop.push(name); + if (state.container && name === DS_NAME) state.container.state = "exited"; + }, + remove: async () => {}, + logs: async () => "", + inspect: async (name: string) => { + calls.inspect.push(name); + return state.inspectBody; + }, + exec: async (name: string, cmd: string[]) => { + calls.exec.push([name, cmd]); + return state.execResult; + }, + adopt: async () => { + throw Object.assign(new Error("no container to adopt"), { status: 404 }); + }, + }, + volumes: { + list: async () => { + calls.volumeList++; + return [...state.volumes]; + }, + create: async (name: string) => { + calls.volumeCreate.push(name); + state.volumes.push({ name, owner: "" }); + }, + remove: async () => {}, + adopt: async () => { + throw Object.assign(new Error("no volume to adopt"), { status: 404 }); + }, + }, + } as unknown as Docker; + return { docker, calls, state }; +} + +async function until(cond: () => boolean, ms = 2000): Promise<void> { + const t0 = Date.now(); + while (!cond()) { + if (Date.now() - t0 > ms) throw new Error("timed out waiting for lookAfter"); + await new Promise((r) => setTimeout(r, 10)); + } +} + +/* ---------------- ensure: the fresh path ---------------- */ + +test("ensure (fresh): pulls image with progress, creates volume + container host-networked with exact JWT env, starts, writes ds.json", async () => { + const root = tmpRoot({ enabled: true, imageTag: TAG92 }); // tag comes from settings, not a constant + const m = makeDocker(); + + const seen: Array<{ active: boolean; line: string }> = []; + const res = await ensureContainer(m.docker, (s) => seen.push({ active: s.active, line: s.line }), root); + assert.deepEqual(res, { started: true }); + + // image ensured exactly once, with the settings tag, and onPull progress captured end-to-end + assert.deepEqual(m.calls.ensure, [TAG92]); + assert.deepEqual(seen, [ + { active: true, line: "downloading layers…" }, + { active: false, line: "done" }, + ]); + + // volume created exactly once, from an empty list + assert.equal(m.calls.volumeList, 1); + assert.deepEqual(m.calls.volumeCreate, [DS_VOLUME]); + assert.ok(m.state.volumes.some((v) => v.name === DS_VOLUME)); + + // container created under the exact name with the spec the contract fixes (no Memory cap -- plan says 4 GB + // recommended, no enforced limit; NetworkMode from sharedNetwork(); the volume bound at the DS data path) + assert.equal(m.calls.create.length, 1); + const [name, spec] = m.calls.create[0]; + assert.equal(name, DS_NAME); + assert.equal(spec.Image, TAG92); + assert.deepEqual(spec.HostConfig, { + NetworkMode: "host", + Binds: [`${DS_VOLUME}:/var/www/onlyoffice/Data`], + }); + // the JWT env is exactly the four documented vars; the secret is the trimmed content of root/jwt-secret + assert.deepEqual(spec.Env, [ + "JWT_ENABLED=true", + "JWT_SECRET=s3cret", + "JWT_HEADER=X-Auth-Token", + "JWT_IN_BODY=false", + ]); + + // started once, by name + assert.deepEqual(m.calls.start, [DS_NAME]); + + // ds.json: id from the create result, host origin, the tag; single source of truth readable back + const ds = JSON.parse(fs.readFileSync(path.join(root, "ds.json"), "utf8")); + assert.deepEqual(ds, { containerId: "cid-1", origin: "http://127.0.0.1", imageTag: TAG92 }); + assert.deepEqual(readDsRecord(root), ds); + + // pull state is module-level in-memory and now surfaced by status() (also: container is running there) + const s = await status(m.docker, root); + assert.equal(s.state, "running"); + assert.deepEqual(s.pull, { active: false, line: "done" }); +}); + +/* ---------------- ensure: adopt path ---------------- */ + +test("ensure (already running): {started:false}, no create/start, volume not re-created, ds.json stays consistent", async () => { + const root = tmpRoot({ enabled: true, imageTag: TAG92 }); + const m = makeDocker(); + await ensureContainer(m.docker, undefined, root); // first pass fills the mock + assert.equal(m.state.container?.state, "running"); + const before = fs.readFileSync(path.join(root, "ds.json"), "utf8"); + + const res = await ensureContainer(m.docker, undefined, root); + assert.deepEqual(res, { started: false }); + assert.equal(m.calls.create.length, 1); // not created again + assert.equal(m.calls.start.length, 1); // and never restarted + assert.equal(m.calls.volumeCreate.length, 1); // volume not duplicated on the second call + assert.equal(m.state.volumes.filter((v) => v.name === DS_VOLUME).length, 1); + assert.equal(fs.readFileSync(path.join(root, "ds.json"), "utf8"), before); // ds.json kept consistent +}); + +test("ensure (container-mode origin): networkMode container:<id> is recorded verbatim in ds.json", async () => { + const root = tmpRoot(); + const m = makeDocker({ networkMode: "container:9f3a1c2d" }); + await ensureContainer(m.docker, undefined, root); + const ds = JSON.parse(fs.readFileSync(path.join(root, "ds.json"), "utf8")); + assert.equal(ds.origin, "container:9f3a1c2d"); // marker string a later wave turns into a URL + const [name, spec] = m.calls.create[0]; + assert.equal(name, DS_NAME); + assert.equal((spec.HostConfig as Record<string, unknown>).NetworkMode, "container:9f3a1c2d"); +}); + +test("ensure (docker unavailable): throws the no_docker error code", async () => { + const root = tmpRoot(); + const m = makeDocker(); + m.state.availableFlag = false; + await assert.rejects(ensureContainer(m.docker, undefined, root), (e: Error & { code?: string }) => e.code === "no_docker"); + assert.deepEqual(m.calls.ensure, []); // nothing was attempted past the guard +}); + +test("ensure (missing jwt-secret): the plain fs error surfaces", async () => { + const root = tmpRoot(undefined, null); // no secret file written + const m = makeDocker(); + await assert.rejects(ensureContainer(m.docker, undefined, root), /ENOENT/); +}); + +/* ---------------- stop + user-stopped intent ---------------- */ + +test("stop: containers.stop called + intent recorded; lookAfter stays quiet; ensure (user start) clears the intent", async () => { + const root = tmpRoot({ enabled: true }); + const m = makeDocker(); + await ensureContainer(m.docker, undefined, root); // now running + + await stopContainer(m.docker, root); + assert.deepEqual(m.calls.stop, [DS_NAME]); + assert.ok(fs.existsSync(path.join(root, "user-stopped")), "intent marker file written"); + assert.equal(userStopped(root), true); + + // lookAfter must NOT ensure/revive while the intent is set (the mock already flipped it to exited on stop) + lookAfter(m.docker, root); + await new Promise((r) => setTimeout(r, 100)); + assert.equal(m.calls.ensure.length, 1); // still only the first pass' pull -- no revive attempt + assert.equal(m.calls.start.length, 1); + + // clearing path: an explicit ensure is a user start request -> starts it and removes the marker + const res = await ensureContainer(m.docker, undefined, root); + assert.deepEqual(res, { started: true }); + assert.equal(userStopped(root), false); + assert.equal(fs.existsSync(path.join(root, "user-stopped")), false); + assert.equal(m.calls.start.length, 2); +}); + +test("stop (no container yet): intent still recorded, no stop call", async () => { + const root = tmpRoot(); + const m = makeDocker(); + await stopContainer(m.docker, root); + assert.deepEqual(m.calls.stop, []); + assert.equal(userStopped(root), true); +}); + +/* ---------------- lookAfter ---------------- */ + +test("lookAfter (settings disabled): no ensure", async () => { + const root = tmpRoot({ enabled: false }); // default + const m = makeDocker(); + lookAfter(m.docker, root); + await new Promise((r) => setTimeout(r, 100)); + assert.deepEqual(m.calls.ensure, []); + assert.deepEqual(m.calls.create, []); +}); + +test("lookAfter (enabled + stopped container): ensures -- pulls, adopts by id, starts", async () => { + const root = tmpRoot({ enabled: true }); + const m = makeDocker(); + // a made-and-exited container by that name: adopted, not recreated + m.state.container = { name: DS_NAME, id: "cid-adopted", image: DEFAULT_TAG, state: "exited", status: "Exited (0)", owner: "", labels: {} }; + + lookAfter(m.docker, root); + await until(() => m.calls.start.length === 1); + assert.deepEqual(m.calls.ensure, [DEFAULT_TAG]); // default settings tag + assert.equal(m.calls.create.length, 0); // existing container reused + const ds = JSON.parse(fs.readFileSync(path.join(root, "ds.json"), "utf8")); + assert.deepEqual(ds, { containerId: "cid-adopted", origin: "http://127.0.0.1", imageTag: DEFAULT_TAG }); +}); + +test("lookAfter (user-stopped): not called at all", async () => { + const root = tmpRoot({ enabled: true }); + fs.writeFileSync(path.join(root, "user-stopped"), ""); + const m = makeDocker(); + lookAfter(m.docker, root); + await new Promise((r) => setTimeout(r, 100)); + assert.deepEqual(m.calls.ensure, []); + assert.equal(userStopped(root), true); // untouched +}); + +/* ---------------- status ---------------- */ + +test("status (absent container): stopped", async () => { + const root = tmpRoot(); + const m = makeDocker(); + const s = await status(m.docker, root); + assert.equal(s.state, "stopped"); + assert.equal(s.version, undefined); + assert.equal(s.memBytes, undefined); +}); + +test("status (running with inspect): running + version from the GitVersion exec probe + memBytes", async () => { + const root = tmpRoot(); + const m = makeDocker(); + m.state.container = { name: DS_NAME, id: "cid-run", image: DEFAULT_TAG, state: "running", status: "Up 3 hours", owner: "", labels: {} }; + m.state.inspectBody = { State: { Status: "running", Running: true }, Memory: 419_430_400 }; + m.state.execResult = { output: JSON.stringify({ DocEditorVersion: "10.0.1" }), exitCode: 0, timedOut: false }; + + const s = await status(m.docker, root); + assert.equal(s.state, "running"); + assert.equal(s.version, "10.0.1"); + assert.equal(s.memBytes, 419_430_400); + // the exact probe: cat of the build file in the DS container + assert.deepEqual(m.calls.exec, [[DS_NAME, ["cat", "/var/www/onlyoffice/documentserver/GitVersion.json"]]]); +}); + +test("status (exited by inspect): stopped; starting flag honored", async () => { + const m = makeDocker(); + m.state.container = { name: DS_NAME, id: "cid-x", image: DEFAULT_TAG, state: "running", status: "Up 0 seconds", owner: "", labels: {} }; + m.state.inspectBody = { State: { Status: "running", Starting: true } }; + const s1 = await status(m.docker); + assert.equal(s1.state, "starting"); + + m.state.inspectBody = { State: { Status: "exited" } }; + const s2 = await status(m.docker); + assert.equal(s2.state, "stopped"); +}); + +test("status (docker unavailable): stopped, no probe attempted", async () => { + const m = makeDocker(); + m.state.availableFlag = false; + const s = await status(m.docker); + assert.equal(s.state, "stopped"); + assert.deepEqual(m.calls.get, []); +}); diff --git a/extensions/office/test/fixtures.test.mts b/extensions/office/test/fixtures.test.mts new file mode 100644 index 000000000..1522012dd --- /dev/null +++ b/extensions/office/test/fixtures.test.mts @@ -0,0 +1,134 @@ +/** + * Cross-check of the zip reader/writer against NON-TRIVIAL committed files (plan 7 "docx fixture + * round-trip", contract tests W5): parse fixtures/sample.docx and fixtures/sample.xlsx with + * ooxml.parseDocx/parseXlsx and assert the exact structure, then edit a quote that SPANS two w:t runs + * in the docx through ooxml.editDocx + rewriteZip and re-parse the output. + * + * The fixtures are binary files committed in this extension's fixtures/ directory; they were generated + * by scripts/make-fixtures.mjs (hand-written OOXML parts zipped through OUR OWN ooxml.writeZip, so these + * tests read back material our own writer produced -- if the reader and writer disagree, that shows here). + * Re-generate with: env -u NODE_ENV node scripts/make-fixtures.mjs (deterministic output). + * jszip is used below ONLY to pull a part's raw bytes out for editDocx input -- parsing and editing go + * through ooxml exclusively, exactly as the runtime paths do. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import { fileURLToPath } from "node:url"; +import JSZip from "jszip"; +import { parseDocx, parseXlsx, editDocx, rewriteZip } from "../src/ooxml"; + +const FIXTURES = fileURLToPath(new URL("../fixtures/", import.meta.url)); +const DOCX = fs.readFileSync(`${FIXTURES}sample.docx`); +const XLSX = fs.readFileSync(`${FIXTURES}sample.xlsx`); + +/** word/document.xml of the docx fixture, pulled with jszip only (the ooxml path has no part-reader export). */ +async function documentXml(): Promise<string> { + const zip = await JSZip.loadAsync(DOCX); + const file = zip.file("word/document.xml"); + if (!file) throw new Error("fixture docx lost its word/document.xml -- regenerate with scripts/make-fixtures.mjs"); + return (await file.async("string")) as string; +} + +/* ---------------- sample.docx ---------------- */ + +test("sample.docx: five paragraphs in order (the third empty), cross-run text joined exactly, entities decoded", () => { + const d = parseDocx(DOCX); + assert.deepEqual(d.paragraphs, [ + // p0: single run; & and </> decoded to raw characters + "Quarterly & annual report for <Meridian Labs>", + // p1: THREE w:t runs joined into one sentence (the cross-run join this fixture exists to prove) + "Revenue figures were revised after the audit found a discrepancy.", + // p2: the deliberately empty paragraph, kept on purpose + "", + // p3: more named entities + "Note: 5 & 7 were checked against the ledger (see <Appendix B>).", + // p4: numeric (—) and hex (€) character references decoded via fromCodePoint + "Footnote: \u2014 total is \u20AC42,300.", + ]); + // `text` is the paragraphs joined with "\n" (parseDocx contract) -- exactly one newline where p2 sits empty + assert.equal(d.text, d.paragraphs.join("\n")); + assert.ok(d.text.includes("discrepancy.\n\nNote:"), "the empty paragraph contributes a blank line to text"); +}); + +test("sample.docx: comments.xml parses two comments from different authors with ids, dates and joined run text", () => { + const d = parseDocx(DOCX); + assert.equal(d.comments.length, 2, "exactly the two comments the fixture carries"); + assert.deepEqual(d.comments[0], { + id: "10", + author: "Alice Chen", + date: "2026-10-25T10:00:00Z", + text: "Please fix & resend the totals.", // two runs joined, & decoded + }); + assert.equal(d.comments[1].id, "25"); + assert.equal(d.comments[1].author, "Bob Ortiz", "second comment has a DIFFERENT author"); + assert.equal(d.comments[1].date, undefined, "the fixture's second comment carries no w:date attribute"); + assert.equal(d.comments[1].text, "@agent the appendix needs re-stating before we ship."); // single run, mention grammar intact +}); + +test("sample.docx: editDocx replaces a quote that SPANS two runs; the rewritten zip still parses and keeps every other part", async () => { + const xml = await documentXml(); + // "revised after the" starts inside run 2 ("were revised after ") and ends inside run 3 + // ("the audit found a discrepancy.") -- a classic Word run-split that naive substring edits corrupt. + const edited = editDocx(xml, [ + { quote: "revised after the", replacement: "walked back before any" }, + // a second op on the RESULT (ops are sequential per contract): single-run target in paragraph 0 + { quote: "Quarterly & annual report", replacement: "Quarterly and annual report" }, + ]); + + const out = rewriteZip(DOCX, { "word/document.xml": edited }); + const after = parseDocx(out); // round-trip through our own reader + assert.deepEqual(after.paragraphs, [ + "Quarterly and annual report for <Meridian Labs>", + "Revenue figures were walked back before any audit found a discrepancy.", + "", + "Note: 5 & 7 were checked against the ledger (see <Appendix B>).", + "Footnote: \u2014 total is \u20AC42,300.", + ]); + // untouched parts survive byte-for-byte in CONTENT (rewriteZip contract), comments included + assert.deepEqual(after.comments, parseDocx(DOCX).comments); + + const beforeParts = await JSZip.loadAsync(DOCX); + const afterParts = await JSZip.loadAsync(out); + assert.deepEqual(Object.keys(afterParts.files).sort(), Object.keys(beforeParts.files).sort(), "no part added or dropped"); + for (const name of Object.keys(beforeParts.files)) { + if (name === "word/document.xml") continue; // the edited part legitimately changed + const a = await beforeParts.file(name)!.async("nodebuffer"); + const b = await afterParts.file(name)!.async("nodebuffer"); + assert.ok(Buffer.compare(a, b) === 0, `untouched part ${name} re-extracts byte-identical`); + } + + // and a quote spanning runs that does not exist in the fixture is refused with ooxml's code + assert.throws(() => editDocx(xml, [{ quote: "revised after THAT", replacement: "" }]), (e: unknown) => { + const err = e as Error & { code?: string }; + return err.code === "quote_not_found"; + }); +}); + +/* ---------------- sample.xlsx ---------------- */ + +test("sample.xlsx: two sheets by name; shared strings (incl. a multi-run <si> with an entity) + inline strings + sparse fill", () => { + const x = parseXlsx(XLSX); + assert.deepEqual(x.sheetNames, ["Inventory", "Notes"]); // workbook order + + // Sheet Inventory: t="s" lookups, plain numbers kept as stored text, an inline string with its padding, + // and a sparse hole (row 3 has A/B/D, no C) filled dense to the farthest used cell. + assert.deepEqual(x.sheets["Inventory"], [ + ["Product", "ACME & Sons Ltd", "12", ""], // B1: shared string with decoded & + ["", "", "", ""], // row 2 absent entirely -> all-empty line in the dense grid + [" on order ", "Grand total (\u00A3)", "", "99"], // A3 inlineStr keeps its spaces; B3 = a TWO-RUN shared string joined, £ decoded + ]); + + // Sheet Notes: one shared-string cell, a decimal kept as stored text, an inline string with double padding. + assert.deepEqual(x.sheets["Notes"], [ + ["Notes sheet", "1.5"], + [" see tab 1 ", ""], + ]); +}); + +test("sample.xlsx: the same bytes still load in jszip -- both zippers agree on the part list (reader sanity)", async () => { + const zip = await JSZip.loadAsync(XLSX); + for (const name of ["[Content_Types].xml", "_rels/.rels", "xl/workbook.xml", "xl/_rels/workbook.xml.rels", "xl/sharedStrings.xml", "xl/worksheets/sheet1.xml", "xl/worksheets/sheet2.xml"]) { + assert.ok(zip.file(name), `fixture xlsx carries ${name}`); + } +}); diff --git a/extensions/office/test/integration.test.mts b/extensions/office/test/integration.test.mts new file mode 100644 index 000000000..696765bd8 --- /dev/null +++ b/extensions/office/test/integration.test.mts @@ -0,0 +1,300 @@ +/** + * Integration tests for pithagoras-office (contract "tests (W5)" / plan 7 "Integration"), each skipped + * with a clear reason when the gate below does not hold. What they assert, exactly as the contract lists: + * + * 1. A ticketed GET on the REAL loopback listener serves a document LARGER than 1 MB byte-for-byte + * (the whole reason the cookie-free listener exists is that portal-managed files cap at MAX_EDIT_BYTES + * = 1 MB -- core/server/src/workspace-files.ts:47) and rejects an expired ticket. + * 2. A synthetic OnlyOffice callback with status 2 arrives on the loopback POST /callback WITHOUT any + * session cookie, persists (bytes on disk changed to exactly what body.url served), bumps the index + * revision, writes history/<doc>/<rev>.zip from changesurl, and clears the lock AFTER the persist. + * 3. office_edit against a doc with a live human session present: the co-editor model means the session + * neither blocks nor gates the edit -- a matching revision succeeds alongside the open editor, and a + * revision that moved underneath (an external co-editor save) surfaces as error:revision_mismatch + * CONFLICT -- nothing is overwritten. + * + * The gate (why it is an env flag + a docker CLI probe instead of host.docker()): the contract says these + * tests are "docker-gated on docker.available()", but src/host.ts' docker() mirrors the portal's + * extensions.use("pithagoras-docker") -- it only exists inside a RUNNING portal process, and this file (like + * every unit test) must never reach portal(). The closest honest probe OUTSIDE a portal is: the operator + * has declared intent via OFFICE_IT_DOCKER=1 AND `docker info` answers from the CLI. When both hold we + * assume the operator pre-started the DS container per the plan's M0 procedure; these three assertions + * themselves run against a tmp dataDir + the real in-process loopback listener, with a tiny second ephemeral + * server standing in for the DS download endpoint (body.url / changesurl), so they exercise exactly the + * HTTP boundary a real DS crosses. The full save-loop-with-real-DS-url and the DocEditor browser assertions + * of plan 7 stay deferred to a Playwright/CI environment with a running portal -- not invented here. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { randomBytes } from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import http from "node:http"; +import { fileURLToPath } from "node:url"; +import type { AddressInfo } from "node:net"; +import type { ServerPortal } from "@pithagoras/extension-api/server"; +import officeServer, { jwtSecret, loopbackPort } from "../src/server"; +import * as store from "../src/store"; +import * as live from "../src/live"; +import { mintTicket } from "../src/tickets"; +import { signJwt } from "../src/jwt"; +import { makeKey } from "../src/callback"; +import { parseDocx, rewriteZip } from "../src/ooxml"; +import { registerTools, __resetToolsForTests } from "../src/tools"; + +const FIXTURES = fileURLToPath(new URL("../fixtures/", import.meta.url)); +/** The committed sample docx doubles as the "what the DS saved" payload in test 2: it differs from the + * blank store template on disk, so "bytes changed" is a meaningful assertion. */ +const SAVED_DOCX = fs.readFileSync(`${FIXTURES}sample.docx`); + +/* ------------------------------------------------------------------ */ +/* the docker gate */ +/* ------------------------------------------------------------------ */ + +/** Reason every test below must be skipped, or undefined when the gate holds. */ +function gateReason(): string | undefined { + if (process.env.OFFICE_IT_DOCKER !== "1") { + return 'docker-gated integration test: set OFFICE_IT_DOCKER=1 to run it (plan M0 procedure -- the operator starts the DS container first; host.docker() is unusable outside a running portal process)'; + } + const probe = spawnSync("docker", ["info"], { encoding: "utf8", timeout: 20_000 }); + if (probe.error) return `docker-gated: the docker CLI is unreachable (${(probe.error as Error).code ?? (probe.error as Error).message})`; + if (probe.status !== 0) return `docker-gated: "docker info" exited ${probe.status}`; + return undefined; // a working docker host exists -- proceed +} + +const SKIP = gateReason(); +/** Register an integration test; node:test reports it as skipped with the reason when SKIP is set. */ +function IT(name: string, fn: () => Promise<void>): void { + test(name, { skip: SKIP }, fn); +} + +/* ------------------------------------------------------------------ */ +/* booting the real server half against a tmp dataDir + fake portal */ +/* ------------------------------------------------------------------ */ + +interface Booted { + root: string; + logs: string[]; + audits: Array<Record<string, unknown>>; + cleanup: () => Promise<void>; +} + +/** The REAL officeServer entry (routes registered on a no-op router, the REAL loopback listener bound to an + * ephemeral 127.0.0.1 port) over a tmp dataDir and a recording fake portal. The fake Docker literal answers + * available()=false so lookAfter never tries to touch anything; settings default disabled keeps it quiet. */ +async function boot(): Promise<Booted> { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-it-")); + const logs: string[] = []; + const audits: Array<Record<string, unknown>> = []; + const fakeDocker = { available: async () => false } as Record<string, unknown>; + const portalObj = { + dataDir: () => root, + log: (l: unknown) => logs.push(String(l)), + audit: { record: (e: Record<string, unknown>) => audits.push(e) }, + router: { get() {}, post() {}, put() {}, delete() {} }, // UI routes are covered by server.test.mts; the listener is what this file drives + http: { json: () => (_a?: unknown, _b?: unknown, next?: () => void) => next?.(), text: () => (_a?: unknown, _b?: unknown, next?: () => void) => next?.(), raw: () => () => {}, send: () => () => {}, upgrade: () => undefined }, + folders: { list: () => [] as never[], allowed: () => false, label: () => ({ name: "" }) }, + chats: { extend: () => {}, get: () => undefined, list: () => [], inFolder: () => [] as never[], folder: () => { throw new Error("no workspace folder"); }, send: () => {}, tell: async () => false }, + extensions: { use: () => fakeDocker }, + } as unknown as ServerPortal; + const cleanup = await officeServer(portalObj); + return { root, logs, audits, cleanup }; +} + +function closeServer(server: http.Server): Promise<void> { + server.closeAllConnections(); // drop undici keep-alives so close() actually finishes + return new Promise((resolve) => server.close(() => resolve())); +} + +/* ------------------------------------------------------------------ */ +/* 1. ticketed GET > 1 MB */ +/* ------------------------------------------------------------------ */ + +IT("ticketed GET on the real loopback listener serves a >1 MB document byte-for-byte and rejects an expired ticket", async () => { + const b = await boot(); + try { + const doc = await store.createDoc({ title: "Heavy Report", type: "word" }, b.root); + + // A REAL .docx over 1 MB, built through this extension's own writer (rewriteZip of the store template). + // Most of the size is one incompressible run: deflate would otherwise shrink repetitive filler below the + // cap and the test would stop proving what it claims. The document still opens/parses -- parseDocx below. + const blob = randomBytes(1_300_000).toString("base64"); // 1.7 MB of text that survives deflate (random bytes -> base64) + const filler = Array.from({ length: 200 }, (_, i) => `<w:p><w:r><w:t xml:space="preserve">filler line ${i + 1} of the heavy integration fixture</w:t></w:r></w:p>`).join(""); + const bigXml = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body><w:p><w:r><w:t>Heavy document</w:t></w:r></w:p>${filler}<w:p><w:r><w:t xml:space="preserve">${blob}</w:t></w:r></w:p></w:body></w:document>`; + const big = rewriteZip(await store.templateDocx(), { "word/document.xml": bigXml }); + assert.ok(big.length > 1024 * 1024, `the built document is ${big.length} bytes (>1 MB)`); + + store.persistBytes(doc, big, b.root); // the meta we hold still matches -- straight through the gate + const fresh = store.getDoc(doc.docId, b.root)!; + assert.ok(fresh.sizeBytes > 1024 * 1024, `the index sizeBytes is ${fresh.sizeBytes} (>1 MB)`); + + const secret = jwtSecret(b.root); + const { token } = mintTicket(fresh.docId, 600_000, secret); + // node's fetch carries no cookie jar: this request -- like the DS's own GET -- crosses without a session cookie. + const r = await fetch(`http://127.0.0.1:${loopbackPort()}/doc/${token}`); + assert.equal(r.status, 200); + assert.equal(r.headers.get("content-type"), "application/vnd.openxmlformats-officedocument.wordprocessingml.document"); + assert.equal(r.headers.get("cache-control"), "no-store"); + const served = Buffer.from(await r.arrayBuffer()); + assert.ok(served.length > 1024 * 1024, `the wire payload is ${served.length} bytes -- past the portal-files 1 MB cap`); + const onDisk = fs.readFileSync(path.join(b.root, "docs", fresh.docId, "Heavy Report.docx")); + assert.equal(Buffer.compare(served, onDisk), 0, "the served bytes are exactly the library bytes"); + // and it is a real Word document end to end, not just bytes: + assert.ok(parseDocx(served).text.startsWith("Heavy document"), "the heavy doc still parses as docx"); + + const { token: dead } = mintTicket(fresh.docId, -1_000, secret); // negative ttl -> already expired at mint time + const r2 = await fetch(`http://127.0.0.1:${loopbackPort()}/doc/${dead}`); + assert.equal(r2.status, 401); + assert.deepEqual(await r2.json(), { error: "invalid or expired document ticket", code: "bad_ticket" }); + + live.clearDoc(fresh.docId); + } finally { + await b.cleanup(); + } +}); + +/* ------------------------------------------------------------------ */ +/* 2. synthetic status-2 callback without a session cookie */ +/* ------------------------------------------------------------------ */ + +IT("synthetic status-2 callback WITHOUT any session cookie: persist happened (bytes changed, index bumped, history zip written) and the lock cleared", async () => { + const b = await boot(); + try { + // The listener's POST /callback was configured at boot to read this tmp root's ds.json + jwt-secret on + // every request -- so writing the record here is what turns the endpoint from 503 into live. + fs.writeFileSync(path.join(b.root, "ds.json"), JSON.stringify({ containerId: "it-cid", origin: "http://127.0.0.1", imageTag: "onlyoffice/documentserver:10.0" })); + const doc = await store.createDoc({ title: "Round Trip", type: "word" }, b.root); + + // A stand-in for the DS download endpoint (body.url / changesurl): a second ephemeral loopback server + // serving the "saved" bytes (our committed fixture docx -- different from the on-disk template) and a + // fixed changes archive. This is exactly the URL pair a real DS callback would carry. + const changesArchive = Buffer.concat([Buffer.from("PK\x03\x04"), randomBytes(1024)]); + let hits: string[] = []; + const dsStandin = http.createServer((req, res) => { + if (req.method === "GET" && req.url === "/saved") { + hits.push("saved"); + res.writeHead(200, { "Content-Type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document" }); + res.end(SAVED_DOCX); + return; + } + if (req.method === "GET" && req.url === "/changes.zip") { + hits.push("changes.zip"); + res.writeHead(200, { "Content-Type": "application/zip" }); + res.end(changesArchive); + return; + } + res.writeHead(404); + res.end("nope"); + }); + await new Promise<void>((resolve, reject) => { dsStandin.once("error", reject); dsStandin.listen(0, "127.0.0.1", () => resolve()); }); + const dsPort = (dsStandin.address() as AddressInfo).port; + + // The state a status-1 join would have left: one human present, the lock flag set. + live.setUser(doc.docId, { id: "u-sam", name: "Sam" }, true); + live.setLocked(doc.docId, true); + const before = store.getDoc(doc.docId, b.root)!; + + // The DS envelope, signed with THIS tmp root's boot secret -- the same value server.ts created on disk. + const secret = jwtSecret(b.root); + const envelope = { + key: makeKey(before.docId, before.mtimeMs), // the only key shape this package mints + status: 2, // ready for saving: terminal close + url: `http://127.0.0.1:${dsPort}/saved`, + changesurl: `http://127.0.0.1:${dsPort}/changes.zip`, + }; + const token = signJwt(envelope, 60_000, secret); + + // POSTed to the loopback listener exactly as DS would (X-Auth-Token per the container env) -- node's fetch + // sends no cookies, so structurally NOTHING like a session cookie crosses on this request. + const r = await fetch(`http://127.0.0.1:${loopbackPort()}/callback`, { + method: "POST", + headers: { "content-type": "application/json", "x-auth-token": token }, + body: JSON.stringify(envelope), + }); + assert.equal(r.status, 200); + assert.deepEqual(await r.json(), { error: 0 }, "DS reads the JSON error field; 0 is the only success"); + + const after = store.getDoc(before.docId, b.root)!; + const onDisk = fs.readFileSync(path.join(b.root, "docs", before.docId, "Round Trip.docx")); + assert.equal(Buffer.compare(onDisk, SAVED_DOCX), 0, "the bytes the callback downloaded are now the library bytes"); + assert.ok(parseDocx(onDisk).comments.length === 2, "and they really ARE a docx -- our own parser reads it back"); + assert.notEqual(after.mtimeMs, before.mtimeMs, "the index revision (mtimeMs) bumped with the persist"); + assert.equal(after.sizeBytes, SAVED_DOCX.length); + const historyZip = path.join(b.root, "history", before.docId, `${after.mtimeMs}.zip`); + assert.ok(fs.existsSync(historyZip), `the changes archive is stored at history/<doc>/<rev>.zip (${historyZip})`); + assert.equal(Buffer.compare(fs.readFileSync(historyZip), changesArchive), 0, "the stored archive is exactly what changesurl served"); + assert.deepEqual(hits.sort(), ["changes.zip", "saved"], "the listener fetched BOTH urls through the loopback path"); + assert.deepEqual(live.usersOf(before.docId), [], "presence cleared by the status-2 terminal close"); + assert.equal(live.isLocked(before.docId), false, "the lock flag cleared AFTER the persist landed"); + + await closeServer(dsStandin); + } finally { + live.resetLive(); + await b.cleanup(); + } +}); + +/* ------------------------------------------------------------------ */ +/* 3. co-editor model: open session + revision gate */ +/* ------------------------------------------------------------------ */ + +IT("office_edit with a live human session: proceeds under the revision gate (matching revision succeeds); an external save that moves the revision underneath is refused as a CONFLICT", async () => { + __resetToolsForTests(); // this file's process starts clean; be explicit + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-it-tools-")); + const home = "chat-aegis-home"; + + // Register through the REAL tools seams: an agent folder whose home chat owns what it edits. + type ToolDef = { name: string; execute: (callId: unknown, params: Record<string, unknown>) => Promise<unknown> }; + const tools2: Record<string, ToolDef> = {}; + registerTools({ + root, + extend: (factory) => factory({ registerTool: (def: ToolDef) => { tools2[def.name] = def; } }, { id: home }), + chatFolder: () => "/workspaces/aegis", + folders: () => [{ path: "/workspaces/aegis", name: "aegis", kind: "agent" }], + inFolder: () => [{ id: home }], + record: () => {}, log: () => {}, + }); + assert.ok(tools2.office_edit, "office_edit registered into the agent home (ownership scope is real here)"); + + /* Scenario A -- the open session must NEITHER block NOR gate: a matching revision succeeds alongside it. */ + const docA = await store.createDoc({ title: "Order Doc", type: "word", owner: home }, root); // template says "New document" + live.setUser(docA.docId, { id: "u-sam", name: "Sam" }, true); // a fake human with the editor open + const fileA = path.join(root, "docs", docA.docId, "Order Doc.docx"); + // Pin the file mtime to an exact value AND mirror it in the index so the ZERO-tolerance gate is deterministic + // (a fresh create can leave a sub-ms stat drift on some filesystems; this guest fs also reports utimes'd + // values ~1 µs low, so the index mirrors what stat ACTUALLY reads back). + fs.utimesSync(fileA, new Date(Date.now() - 10_000), new Date(Date.now() - 10_000)); + const baseA = fs.statSync(fileA).mtimeMs; + { const ip = path.join(root, "index.json"); const raw = JSON.parse(fs.readFileSync(ip, "utf8")); raw.docs.find((d: { docId?: string }) => d.docId === docA.docId).mtimeMs = baseA; fs.writeFileSync(ip, JSON.stringify(raw, null, 2)); } + await tools2.office_edit.execute(null, { id: docA.docId, ops: [{ quote: "New document", replacement: "Edited by the agent" }], baseRevision: baseA }); + assert.ok(parseDocx(fs.readFileSync(fileA)).text.includes("Edited by the agent"), "the edit landed at a new revision while the human session is still open"); + assert.notEqual(store.getDoc(docA.docId, root)!.mtimeMs, docA.mtimeMs); + + /* Scenario B -- an EXTERNAL co-editor save (persistBytes, the same path the DS callback persist takes) + * moves the revision after the agent read it: office_edit must refuse error:revision_mismatch CONFLICT + * and leave the human's bytes on disk. */ + const docB = await store.createDoc({ title: "Bump Doc", type: "word", owner: home }, root); + live.setUser(docB.docId, { id: "u-sam", name: "Sam" }, true); + const fileB = path.join(root, "docs", docB.docId, "Bump Doc.docx"); + // Same pin as Scenario A: the agent reads at exactly this revision (index mirrors the stat readback)... + fs.utimesSync(fileB, new Date(Date.now() - 8000), new Date(Date.now() - 8000)); + const baseB = fs.statSync(fileB).mtimeMs; + { const ip = path.join(root, "index.json"); const raw = JSON.parse(fs.readFileSync(ip, "utf8")); raw.docs.find((d: { docId?: string }) => d.docId === docB.docId).mtimeMs = baseB; fs.writeFileSync(ip, JSON.stringify(raw, null, 2)); } + await new Promise((r) => setTimeout(r, 5)); // guarantee >1 ms of drift: mtime-as-revision moves for real + store.persistBytes(store.getDoc(docB.docId, root)!, SAVED_DOCX, root); // ...a human co-editor's save lands a NEWER revision under our feet + let thrown: unknown; + try { + await tools2.office_edit.execute(null, { id: docB.docId, ops: [{ quote: "New document", replacement: "should never land" }], baseRevision: baseB }); + } catch (e) { + thrown = e; + } + assert.ok(thrown instanceof Error, "office_edit refused"); + assert.match((thrown as Error).message, /^error:revision_mismatch /, `the refusal carries the coded prefix (got: ${(thrown as Error).message})`); + assert.match((thrown as Error).message, /CONFLICT/, "the refusal is an explicit conflict message"); + assert.equal(Buffer.compare(fs.readFileSync(fileB), SAVED_DOCX), 0, "the HUMAN's save is what stayed on disk; the agent edit never wrote"); + + __resetToolsForTests(); + live.resetLive(); +}); diff --git a/extensions/office/test/jszip.test.mts b/extensions/office/test/jszip.test.mts new file mode 100644 index 000000000..7cf88b4c8 --- /dev/null +++ b/extensions/office/test/jszip.test.mts @@ -0,0 +1,40 @@ +/** + * Unit tests for src/jszip.ts: the LAZY jszip loader that replaces the old static `import JSZip from "jszip"` -- + * the built server.js must import cleanly without this package's own node_modules yet, and only the first + * template/import call waits for deps.ensureDependencies (wired through armJsZipInstall at boot). Assertions: + * one load per process, the armed install promise gates that first load, and the result really is jszip. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { jsZip, armJsZipInstall } from "../src/jszip"; + +test("jsZip gates its first load on the armed install promise, then resolves to the real package", async () => { + let release: () => void; + const gate = new Promise<void>((resolve) => (release = resolve)); + armJsZipInstall(gate.then(() => true)); // what server.ts arms with is ensureDependencies' promise + + const load = jsZip(); + let settled = false; + load.then( + () => (settled = true), + () => (settled = true), + ); + await new Promise((r) => setTimeout(r, 25)); // the install has not run: the load must still be pending + assert.equal(settled, false, "jsZip must wait for the armed install before importing jszip"); + + release(); + const zip = await load; + assert.equal(typeof (zip as unknown as { loadAsync?: unknown }).loadAsync, "function", "the static loadAsync must be there"); + const instance = new (zip as unknown as { new (): { file: unknown } })() as unknown as { file: unknown }; + assert.equal(typeof instance.file, "function", "a bare new() must give a working archive"); + + // One load per process: the next call returns the very same promise's result without another gate. + armJsZipInstall(new Promise(() => {})); // never settles: would hang again if re-gated + assert.equal(await jsZip(), zip, "the cached load is returned; a later arm does not re-load"); +}); + +test("an UNARMD install goes straight to the import (unit tests never boot the server entry)", async () => { + // Fresh process for this file means the module state above is the only arming: assert the resolved shape one more time. + const zip = await jsZip(); + assert.equal(typeof (zip as unknown as { version?: string }).version, "string", "jszip exposes its version string"); +}); diff --git a/extensions/office/test/jwt-tickets.test.mts b/extensions/office/test/jwt-tickets.test.mts new file mode 100644 index 000000000..d8ac469b3 --- /dev/null +++ b/extensions/office/test/jwt-tickets.test.mts @@ -0,0 +1,142 @@ +/** + * Unit tests for src/jwt.ts and src/tickets.ts: sign/verify happy paths, tampered payload/signature/header-alg + * (RS256, "none", truncated) all -> null, expired tokens -> null, ttlMs undefined -> no exp claim; ticket + * mint/verify round-trip, expiry, docId flip, wrong secret. Pure crypto only -- no fs, no portal, secrets are + * test literals. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { createHmac } from "node:crypto"; +import { signJwt, verifyJwt } from "../src/jwt"; +import { mintTicket, verifyTicket } from "../src/tickets"; + +const SECRET = "unit-test-secret-0123456789abcdef0123456789abcdef0123456789abcdef01234567"; + +function b64url(text: string): string { + return Buffer.from(text, "utf8").toString("base64url"); +} + +/* ---------------- jwt ---------------- */ + +test("jwt: sign/verify round-trip keeps the payload and adds iat/exp", () => { + const token = signJwt({ sub: "doc-123", role: "editor" }, 60_000, SECRET); + const payload = verifyJwt(token, SECRET); + assert.ok(payload); + assert.equal(payload.sub, "doc-123"); + assert.equal(payload.role, "editor"); + assert.equal(typeof payload.iat, "number"); + assert.equal(typeof payload.exp, "number"); + // seconds precision: exp is within [now+60, now+61] + const nowSec = Math.floor(Date.now() / 1000); + assert.ok(payload.exp! >= nowSec + 60 && payload.exp! <= nowSec + 62, `exp ${payload.exp} not ~now+60`); +}); + +test("jwt: ttlMs undefined -> no iat/exp claims, still verifies", () => { + const token = signJwt({ sub: "x" }, undefined, SECRET); + const payload = verifyJwt(token, SECRET); + assert.deepEqual(payload, { sub: "x" }); // exactly the original claims -- no iat, no exp +}); + +test("jwt: expired token -> null", () => { + const token = signJwt({ sub: "x" }, -5000, SECRET); // signed 5s in the past + assert.equal(verifyJwt(token, SECRET), null); +}); + +test("jwt: tampered payload -> null", () => { + const token = signJwt({ sub: "x" }, 60_000, SECRET); + const [head, body, sig] = token.split("."); + // bump one claim and keep the original signature + const claims = JSON.parse(Buffer.from(body, "base64url").toString("utf8")); + claims.sub = "evil"; + assert.equal(verifyJwt(`${head}.${b64url(JSON.stringify(claims))}.${sig}`, SECRET), null); +}); + +test("jwt: tampered signature -> null (incl. truncated/garbage)", () => { + const token = signJwt({ sub: "x" }, 60_000, SECRET); + const [head, body, sig] = token.split("."); + assert.equal(verifyJwt(`${head}.${body}.${sig.slice(0, -3)}AAAAAA`, SECRET), null); + assert.equal(verifyJwt(`${head}.${body}.not-base64-!!!`, SECRET), null); + assert.equal(verifyJwt(`${head}.${body}`, SECRET), null); // missing segment +}); + +test("jwt: header alg RS256 -> null even with a valid HMAC for the fake header", () => { + const head = b64url(JSON.stringify({ alg: "RS256", typ: "JWT" })); + const body = b64url(JSON.stringify({ sub: "x", exp: Math.floor(Date.now() / 1000) + 60 })); + // a perfectly valid HMAC over the RS256 header -- must still be rejected (alg is pinned, not just trusted) + const sig = createHmac("sha256", SECRET).update(`${head}.${body}`).digest("base64url"); + assert.equal(verifyJwt(`${head}.${body}.${sig}`, SECRET), null); +}); + +test("jwt: alg 'none' -> null; wrong secret -> null; garbage token -> null", () => { + const body = b64url(JSON.stringify({ sub: "x" })); + const noneHead = b64url(JSON.stringify({ alg: "none", typ: "JWT" })); + assert.equal(verifyJwt(`${noneHead}.${body}.`, SECRET), null); + const good = signJwt({ sub: "x" }, 60_000, SECRET); + assert.equal(verifyJwt(good, "a different secret entirely"), null); + assert.equal(verifyJwt("not-a-token", SECRET), null); +}); + +/* ---------------- tickets ---------------- */ + +test("tickets: mint/verify round-trip returns the docId", () => { + const { token, expiresMs } = mintTicket("my-doc-abc123", undefined, SECRET); + assert.ok(token && typeof token === "string"); + assert.ok(expiresMs > Date.now() + 590_000 && expiresMs <= Date.now() + 600_000); // ~10 minute default + assert.deepEqual(verifyTicket(token, SECRET), { docId: "my-doc-abc123" }); +}); + +test("tickets: expired ticket -> null", () => { + const { token } = mintTicket("my-doc", -5000, SECRET); // already in the past + assert.equal(verifyTicket(token, SECRET), null); +}); + +test("tickets: flipped character in the docId section -> null", () => { + const { token } = mintTicket("doc-a-1", 60_000, SECRET); + const decoded = Buffer.from(token, "base64url").toString("utf8"); + const [docId, expMs, mac] = decoded.split("|"); + // a different docId with the SAME expiry and signature must not verify (HMAC covers both sections) + const forged = Buffer.from(`doc-b-2|${expMs}|${mac}`).toString("base64url"); + assert.notEqual(forged, token); + assert.equal(verifyTicket(forged, SECRET), null); +}); + +test("tickets: wrong secret -> null", () => { + const { token } = mintTicket("doc-a-1", 60_000, SECRET); + assert.equal(verifyTicket(token, "wrong-secret"), null); +}); + +test("tickets: malformed tokens -> null (never throws)", () => { + assert.equal(verifyTicket("", SECRET), null); + assert.equal(verifyTicket("!!!not-base64!!!", SECRET), null); // outside the base64url alphabet + assert.equal(verifyTicket(Buffer.from("onlytwo|3").toString("base64url"), SECRET), null); // two sections + assert.equal(verifyTicket(Buffer.from("|1700000000000|abcdef").toString("base64url"), SECRET), null); // empty docId section +}); + +/* ---------------- purpose separation (S3) ---------------- */ + +test("jwt: a signed typ claim separates token families; each verifier pins its own purpose", () => { + const cfg = signJwt({ docId: "d1" }, 60_000, SECRET, "editor-config"); + assert.equal(verifyJwt(cfg, SECRET, "editor-config")?.docId, "d1"); // right family verifies + assert.equal(verifyJwt(cfg, SECRET, "fetch"), null); // an editor-config token is not a fetch token + assert.ok(verifyJwt(cfg, SECRET), "no purpose requested -> accepted (DS-signed callback tokens carry no typ)"); + + const fetch = signJwt({ docId: "d1", key: "k" }, 60_000, SECRET, "fetch"); + assert.equal(verifyJwt(fetch, SECRET, "fetch")?.key, "k"); + assert.equal(verifyJwt(fetch, SECRET, "editor-config"), null); + + // the typ claim is SIGNED: stripping or rewriting it outside the signature breaks verification of that family + const [head, body, sig] = cfg.split("."); + const stripped = JSON.parse(Buffer.from(body, "base64url").toString("utf8")); + delete stripped.typ; + assert.equal(verifyJwt(`${head}.${b64url(JSON.stringify(stripped))}.${sig}`, SECRET, "editor-config"), null); +}); + +test("tickets: the HMAC covers a purpose domain separator -- an un-prefixed MAC does not verify", () => { + const { token } = mintTicket("doc-x-1", 60_000, SECRET); + assert.deepEqual(verifyTicket(token, SECRET), { docId: "doc-x-1" }); // round-trip intact; payload layout unchanged + const [docId, expStr, mac] = Buffer.from(token, "base64url").toString("utf8").split("|"); + // A MAC computed over the legacy/un-prefixed message is a different family's keying and must be rejected: + const unprefixed = createHmac("sha256", SECRET).update(`${docId}|${expStr}`).digest("hex"); + assert.notEqual(unprefixed, mac, "the minted MAC visibly includes the purpose prefix"); + assert.equal(verifyTicket(Buffer.from(`${docId}|${expStr}|${unprefixed}`).toString("base64url"), SECRET), null); +}); diff --git a/extensions/office/test/live.test.mts b/extensions/office/test/live.test.mts new file mode 100644 index 000000000..d062332c1 --- /dev/null +++ b/extensions/office/test/live.test.mts @@ -0,0 +1,69 @@ +/** + * Unit tests for src/live.ts' presence TTL: a tracked (server-minted or heartbeat-fed) user auto-expires after + * the TTL instead of holding the lock forever (S2 stuck-lock DoS); a live session stays by heartbeating; + * co-edit semantics are otherwise unchanged (a fresh open still locks, the explicit flag is independent). + * Pure in-memory module -- no portal, no fs; short real timers only. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { usersOf, setUser, setLocked, isLocked, pollState, clearDoc, resetLive, setPresenceTtl, getPresenceTtl } from "../src/live"; + +const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms)); + +test("presence TTL: a non-heartbeating user auto-expires and stops locking", async () => { + resetLive(); + setPresenceTtl(50); + setUser("doc-a", { id: "portal-user", name: "Portal" }, true); + assert.equal(usersOf("doc-a").length, 1, "freshly minted presence locks immediately"); + assert.equal(pollState("doc-a", 0).locked, true); + + await sleep(80); // past the 50 ms TTL with no heartbeat + assert.deepEqual(usersOf("doc-a"), [], "the stuck presence expired (no terminal close needed)"); + assert.equal(pollState("doc-a", 0).locked, false, "expired presence must not keep refusing agent edits"); + + resetLive(); +}); + +test("presence TTL: every status-1 heartbeat pushes the expiry forward, a live session never trips it", async () => { + resetLive(); + setPresenceTtl(120); + setUser("doc-b", { id: "portal-user", name: "Portal" }, true); + const first = usersOf("doc-b")[0]; + + await sleep(70); + setUser("doc-b", { id: "portal-user", name: "Portal" }, true); // heartbeat re-sets expiresAt (but keeps `at`) + const second = usersOf("doc-b")[0]; + assert.equal(second.at, first.at, "the badge still shows the FIRST join time"); + + await sleep(70); // 140 ms since the original mint (> TTL), but only 70 ms since the heartbeat + assert.equal(usersOf("doc-b").length, 1, "heartbeats keep a live session present past its original TTL"); + assert.equal(pollState("doc-b", 0).locked, true); + + await sleep(80); // now past even the heartbeated expiry + assert.deepEqual(usersOf("doc-b"), [], "and it still expires once heartbeats stop"); + resetLive(); +}); + +test("TTL settings: setPresenceTtl rejects nonsense values; default is 15 minutes (the config signature lifetime)", () => { + resetLive(); + assert.equal(getPresenceTtl(), 15 * 60_000, "default TTL matches CONFIG_TTL_MS (editor-config signature lifetime)"); + setPresenceTtl(0); + setPresenceTtl(-5); + setPresenceTtl(Number.NaN); + assert.equal(getPresenceTtl(), 15 * 60_000, "non-positive/NaN TTLs are ignored rather than stored"); + resetLive(); // also restores the default after a real change +}); + +test("co-edit semantics unchanged: explicit lock flag is independent of presence; clearDoc still clears both", () => { + resetLive(); + setLocked("doc-c", true); // e.g. a status-6 "keep locked" with no tracked user at all + assert.equal(usersOf("doc-c").length, 0); + assert.equal(isLocked("doc-c"), true); + assert.equal(pollState("doc-c", 0).locked, true, "the explicit flag locks on its own"); + + setUser("doc-d", { id: "portal-user" }, true); + clearDoc("doc-d"); // the DELETE /documents/:id path -- presence AND flag gone at once + assert.deepEqual(usersOf("doc-d"), []); + assert.equal(isLocked("doc-d"), false); + resetLive(); +}); diff --git a/extensions/office/test/office-ui.test.mts b/extensions/office/test/office-ui.test.mts new file mode 100644 index 000000000..6964d6bd1 --- /dev/null +++ b/extensions/office/test/office-ui.test.mts @@ -0,0 +1,35 @@ +/** + * The browser half's pure helpers (src/office-ui.ts): where a page address says its opened document, and the + * client-side mirror of the server's floating-image-tag refusal. Plain node: no portal, no DOM. + */ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { editorsIn, floatingTag, parseDocQuery } from "../src/office-ui"; + +test("parseDocQuery: ?doc=<id> is the opened document; encoded ids come back decoded", () => { + assert.equal(parseDocQuery("?doc=my-report"), "my-report"); + assert.equal(parseDocQuery("?doc=report%20one&x=1"), "report one"); + assert.equal(parseDocQuery("/office?doc=a-b-c_123"), "a-b-c_123"); +}); + +test("parseDocQuery: no doc, an empty one, or other parameters mean the list view", () => { + assert.equal(parseDocQuery(""), null); + assert.equal(parseDocQuery("?folder=9&doc="), null); + assert.equal(parseDocQuery("?x=1"), null); +}); + +test("floatingTag: the server's :latest/:stable/@latest refusal, mirrored in the input", () => { + assert.equal(floatingTag("onlyoffice/documentserver:latest"), true); + assert.equal(floatingTag("onlyoffice/documentserver:LATEST"), true); + assert.equal(floatingTag("onlyoffice/documentserver:stable"), true); + assert.equal(floatingTag("registry.example/onlyoffice@latest"), true); + assert.equal(floatingTag("onlyoffice/documentserver:10.0"), false); + assert.equal(floatingTag("onlyoffice/documentserver"), false); +}); + +test("editorsIn: every doc has a locks entry (empty when nobody is in it); missing maps say none", () => { + const locks = { a: [], b: [{ id: "u1", name: "n" }, { id: "u2" }] }; + assert.equal(editorsIn(locks, "a"), 0); + assert.equal(editorsIn(locks, "b"), 2); + assert.equal(editorsIn(undefined, "a"), 0); +}); diff --git a/extensions/office/test/ooxml.test.mts b/extensions/office/test/ooxml.test.mts new file mode 100644 index 000000000..e13a60cd6 --- /dev/null +++ b/extensions/office/test/ooxml.test.mts @@ -0,0 +1,311 @@ +/** + * Unit tests for src/ooxml.ts: docx/xlsx parsing, quote-based document editing, zip rewriting, + * comment diffing. Self-contained -- fixtures are built in-test with jszip (no network, docker, portal). + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import JSZip from "jszip"; +import { parseDocx, parseXlsx, editDocx, rewriteZip, diffComments, type Comment } from "../src/ooxml"; + +/* ---------------- fixture building (jszip) ---------------- */ + +const DOCUMENT_XML = `<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body>` + + `<w:p><w:r><w:t>Hello & welcome to <Office>.</w:t></w:r></w:p>` + + `<w:p><w:r><w:t>Alp</w:t></w:r><w:r><w:t>ha</w:t></w:r><w:r><w:t xml:space="preserve"> beta</w:t></w:r></w:p>` + + `<w:p/>` + + `<w:p><w:r><w:t>Gamma one</w:t></w:r><w:r><w:t xml:space="preserve"> and gamma two</w:t></w:r></w:p>` + + `</w:body></w:document>`; + +const COMMENTS_XML = `<w:comments xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">` + + `<w:comment w:id="1" w:author="Alice Adams" w:date="2026-10-25T10:00:00Z"><w:p><w:r><w:t>Fix the</w:t></w:r><w:r><w:t> title.</w:t></w:r></w:p></w:comment>` + + `<w:comment w:id="2" w:author="Bob Brown"><w:p><w:r><w:t>@agent</w:t></w:r><w:r><w:t> please review</w:t></w:r></w:p></w:comment>` + + `</w:comments>`; + +const CONTENT_TYPES = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Default Extension="xml" ContentType="application/xml"/><Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/><Override PartName="/word/comments.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.comments+xml"/></Types>`; + +const ROOT_RELS = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/></Relationships>`; + +async function buildDocxZip(documentXml = DOCUMENT_XML, commentsXml: string | null = COMMENTS_XML): Promise<Buffer> { + const zip = new JSZip(); + zip.file("[Content_Types].xml", CONTENT_TYPES); + zip.file("_rels/.rels", ROOT_RELS); + zip.file("word/document.xml", documentXml); + if (commentsXml !== null) zip.file("word/comments.xml", commentsXml); + return zip.generateAsync({ type: "nodebuffer" }); +} + +const WORKBOOK_XML = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"><sheets><sheet name="Sales" sheetId="1" r:id="rId1"/><sheet name="Notes" sheetId="2" r:id="rId2"/></sheets></workbook>`; + +const WORKBOOK_RELS = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet1.xml"/><Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet2.xml"/></Relationships>`; + +const SHARED_STRINGS = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><sst xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" count="4" uniqueCount="4"><si><t>Region</t></si><si><t>North</t></si><si><r><t>So</t></r><r><t>uth & Co.</t></r></si><si><t>Total</t></si></sst>`; + +const SALES_SHEET = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData><row r="1"><c r="A1" t="s"><v>0</v></c><c r="B1" t="s"><v>1</v></c><c r="C1"><v>42</v></c></row><row r="2"><c r="A2" t="s"><v>2</v></c><c r="B2" t="inlineStr"><is><t xml:space="preserve"> inline value </t></is></c><c r="C2"><v>7.5</v></c></row></sheetData></worksheet>`; + +// sparse on purpose: A1 and B5 only -- the grid must be filled dense to 5 rows x 2 columns +const NOTES_SHEET = `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData><row r="1"><c r="A1"><v>1</v></c></row><row r="5"><c r="B5" t="inlineStr"><is><t>far cell</t></is></c></row></sheetData></worksheet>`; + +async function buildXlsxZip(): Promise<Buffer> { + const zip = new JSZip(); + zip.file("[Content_Types].xml", CONTENT_TYPES); + zip.file("_rels/.rels", ROOT_RELS); + zip.file("xl/workbook.xml", WORKBOOK_XML); + zip.file("xl/_rels/workbook.xml.rels", WORKBOOK_RELS); + zip.file("xl/sharedStrings.xml", SHARED_STRINGS); + zip.file("xl/worksheets/sheet1.xml", SALES_SHEET); + zip.file("xl/worksheets/sheet2.xml", NOTES_SHEET); + return zip.generateAsync({ type: "nodebuffer" }); +} + +/** Uncompressed bytes of every file entry, keyed by name (jszip on both sides makes comparisons independent of our writer). */ +async function entryBytes(buf: Buffer): Promise<Record<string, Buffer>> { + const zip = await JSZip.loadAsync(buf); + const out: Record<string, Buffer> = {}; + for (const [name, file] of Object.entries(zip.files)) if (!file.dir) out[name] = Buffer.from(await file.async("nodebuffer")); + return out; +} + +/** Run fn, require it to throw, and return the thrown error (this node's assert.throws does not return it). */ +function thrown(fn: () => unknown): Error & { code?: string } { + try { fn(); } catch (err) { if (err instanceof Error) return err as Error & { code?: string }; throw err; } + throw new assert.AssertionError({ message: "expected the call to throw" }); +} + +/* ---------------- parseDocx ---------------- */ + +test("parseDocx extracts paragraphs, joined text and comments with entity decoding", async () => { + const doc = parseDocx(await buildDocxZip()); + assert.deepEqual(doc.paragraphs, [ + "Hello & welcome to <Office>.", + "Alpha beta", + "", // the empty paragraph is kept, on purpose + "Gamma one and gamma two", + ]); + assert.equal(doc.text, doc.paragraphs.join("\n")); + assert.deepEqual(doc.comments, [ + { id: "1", author: "Alice Adams", date: "2026-10-25T10:00:00Z", text: "Fix the title." }, + { id: "2", author: "Bob Brown", text: "@agent please review" }, // no w:date attribute -> no date field + ]); +}); + +test("parseDocx treats a missing comments part as no comments, and rejects non-docx zips", async () => { + const doc = parseDocx(await buildDocxZip(DOCUMENT_XML, null)); + assert.deepEqual(doc.comments, []); + const plainZip = await new JSZip().file("readme.txt", "no document here").generateAsync({ type: "nodebuffer" }); + assert.equal(thrown(() => parseDocx(plainZip)).code, "missing_part"); +}); + +/* ---------------- parseXlsx ---------------- */ + +test("parseXlsx resolves shared strings and inlineStr across both sheets, dense grid", async () => { + const book = parseXlsx(await buildXlsxZip()); + assert.deepEqual(book.sheetNames, ["Sales", "Notes"]); + assert.deepEqual(book.sheets["Sales"], [ + ["Region", "North", "42"], // shared string, shared string, numeric kept as stored text + ["South & Co.", " inline value ", "7.5"], // rich-text si (two runs) + inlineStr with padding kept + ]); + assert.deepEqual(book.sheets["Notes"], [ + ["1", ""], + ["", ""], + ["", ""], + ["", ""], + ["", "far cell"], // sparse cells filled dense to the farthest used row/column + ]); +}); + +test("parseXlsx bounds the dense grid of sparse far-out cells (memory/CPU cap, honest truncation)", async () => { + // The review's attack shape: a ~1.7 KB xlsx with few cells referencing huge coordinates used to densify to + // row x col PER SHEET and allocate hundreds of MB (+~18 s CPU). Each sheet here holds exactly two cells -- + // A1 and one near the axis caps (row 20000, column 500 = MAX_GRID_COLS) -- so every sheet's PRODUCT is + // ~10M slots; the bounded grid must materialize at most floor(MAX_SHEET_CELLS / 500) rows per sheet and flag it. + const colLetters = (n: number): string => { let s = ""; while (n > 0) { const d = (n - 1) % 26; s = String.fromCharCode(65 + d) + s; n = Math.floor((n - 1) / 26); } return s; }; + const farRef = `${colLetters(500)}20000`; // the 500th column (zero-based 499), row 20000 -- both at the axis caps + const sheetXml = `<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData><row r="1"><c r="A1" t="inlineStr"><is><t>origin</t></is></c></row><row r="20000"><c r="${farRef}" t="inlineStr"><is><t>far away</t></is></c></row></sheetData></worksheet>`; + const zip = new JSZip(); + zip.file("[Content_Types].xml", CONTENT_TYPES); + zip.file("_rels/.rels", ROOT_RELS); + zip.file("xl/workbook.xml", `<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"><sheets>${[1, 2, 3].map((i) => `<sheet name="S${i}" sheetId="${i}" r:id="rId${i}"/>`).join("")}</sheets></workbook>`); + zip.file("xl/_rels/workbook.xml.rels", `<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">${[1, 2, 3].map((i) => `<Relationship Id="rId${i}" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet${i}.xml"/>`).join("")}</Relationships>`); + for (const i of [1, 2, 3]) zip.file(`xl/worksheets/sheet${i}.xml`, sheetXml); + const buf = await zip.generateAsync({ type: "nodebuffer" }); + + const t0 = Date.now(); + const book = parseXlsx(buf); // must stay fast and bounded -- the pre-fix run took ~18 s CPU / ~523 MB heap here + assert.ok(Date.now() - t0 < 2000, `parseXlsx on a tiny sparse xlsx must be fast (took ${Date.now() - t0} ms)`); + assert.deepEqual(book.sheetNames, ["S1", "S2", "S3"]); + for (const name of book.sheetNames) { + const grid = book.sheets[name]; + assert.ok(grid.length > 0, `${name}: the origin row is within budget`); + assert.equal(grid[0][0], "origin", `${name}: A1 readable in the affordable rows`); + assert.ok(grid.length < 20000, `${name}: grid height bounded (got ${grid.length} rows, file references row 20000)`); + assert.equal(book.truncatedSheets[name], true, `${name}: the skipped far rows are reported honestly`); + } +}); + +/* ---------------- editDocx ---------------- */ + +/** editDocx works on the XML string; wrap its result into a docx and reparse it. */ +async function reparsed(documentXml: string, commentsXml: string | null = null) { + return parseDocx(await buildDocxZip(documentXml, commentsXml)); +} + +test("editDocx replaces a quote inside a single run and keeps entities round-tripping", async () => { + const doc = await reparsed(editDocx(DOCUMENT_XML, [{ quote: "Gamma one", replacement: "Delta & nine" }])); + assert.deepEqual(doc.paragraphs[3], "Delta & nine and gamma two"); // the & in the replacement is escaped in XML, decoded on reparse +}); + +test("editDocx spans multiple runs at w:t boundaries", async () => { + // "Alpha beta" lives in runs ["Alp","ha"," beta"]; "lpha be" starts inside run 2 and ends inside run 3. + const doc = await reparsed(editDocx(DOCUMENT_XML, [{ quote: "lpha be", replacement: "middle" }])); + assert.deepEqual(doc.paragraphs[1], "Amiddleta"); // "A" + "middle" + "ta" +}); + +test("editDocx applies ops sequentially -- op two quotes op one's output", async () => { + const doc = await reparsed(editDocx(DOCUMENT_XML, [ + { quote: "Gamma one", replacement: "Delta nine" }, + { quote: "Delta nine", replacement: "Final answer" }, // sees the first edit + ])); + assert.deepEqual(doc.paragraphs[3], "Final answer and gamma two"); + // and a stale op that quotes the pre-edit text fails with quote_not_found: + const err = thrown(() => editDocx(DOCUMENT_XML, [ + { quote: "Gamma one", replacement: "Delta nine" }, + { quote: "Gamma one", replacement: "x" }, + ])); + assert.equal(err.code, "quote_not_found"); +}); + +test("editDocx throws quote_not_found and quote_ambiguous", () => { + const missing = thrown(() => editDocx(DOCUMENT_XML, [{ quote: "not in the document", replacement: "x" }])); + assert.equal(missing.code, "quote_not_found"); + // a quote appearing in two paragraphs must be refused + const duplicated = `<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body>` + + `<w:p><w:r><w:t>alpha</w:t></w:r></w:p><w:p><w:r><w:t>beta alpha omega</w:t></w:r></w:p></w:body></w:document>`; + const ambiguous = thrown(() => editDocx(duplicated, [{ quote: "alpha", replacement: "x" }])); + assert.equal(ambiguous.code, "quote_ambiguous"); +}); + +/* ---------------- rewriteZip ---------------- */ + +test("rewriteZip with no changes round-trips: parse-equal and every entry content-identical", async () => { + const original = await buildDocxZip(); + const rewritten = rewriteZip(original, {}); + assert.deepEqual(parseDocx(rewritten), parseDocx(original)); + const before = await entryBytes(original); + const after = await entryBytes(rewritten); + assert.deepEqual(Object.keys(after).sort(), Object.keys(before).sort()); + for (const name of Object.keys(before)) assert.ok(Buffer.compare(before[name], after[name]) === 0, `entry ${name} content changed`); +}); + +test("rewriteZip replaces word/document.xml in place and reflects it on reparse", async () => { + const original = await buildDocxZip(); + const edited = editDocx(DOCUMENT_XML, [{ quote: "Gamma one", replacement: "Delta nine" }]); + const rewritten = rewriteZip(original, { "word/document.xml": edited }); + assert.equal(parseDocx(rewritten).paragraphs[3], "Delta nine and gamma two"); + assert.deepEqual(parseDocx(rewritten).comments, parseDocx(original).comments); + const before = await entryBytes(original); + const after = await entryBytes(rewritten); + for (const name of Object.keys(before)) { + if (name === "word/document.xml") continue; + assert.ok(Buffer.compare(before[name], after[name]) === 0, `entry ${name} content changed`); + } +}); + +test("rewriteZip appends a new part and leaves the rest untouched", async () => { + const original = await buildDocxZip(); + const rewritten = rewriteZip(original, { "word/media/hello.txt": Buffer.from("a binary \u0000\u0001 part", "latin1") }); + const after = await entryBytes(rewritten); + const before = await entryBytes(original); + assert.equal(after["word/media/hello.txt"].toString("latin1"), "a binary \u0000\u0001 part"); + for (const name of Object.keys(before)) assert.ok(Buffer.compare(before[name], after[name]) === 0, `entry ${name} content changed`); +}); + +test("rewriteZip matches part names case-insensitively (no shadow duplicate) and dedupes case variants last-one-wins", async () => { + const original = await buildDocxZip(); + const beforeNames = Object.keys(await entryBytes(original)).sort(); + + // a key that differs only in case from the existing part must OVERWRITE it in place (original name kept), not shadow it + const edited = editDocx(DOCUMENT_XML, [{ quote: "Gamma one", replacement: "Case fixed" }]); + const out1 = rewriteZip(original, { "WORD/DOCUMENT.XML": edited }); + assert.deepEqual(Object.keys(await entryBytes(out1)).sort(), beforeNames, "no shadow part was added"); + assert.equal(parseDocx(out1).paragraphs[3], "Case fixed and gamma two"); + + // two change keys differing only in case dedupe to the LAST one (jszip last-one-wins), still exactly one part + const other = editDocx(DOCUMENT_XML, [{ quote: "Gamma one", replacement: "Last wins" }]); + const out2 = rewriteZip(original, { "word/document.xml": edited, "WORD/DOCUMENT.XML": other }); + assert.equal(parseDocx(out2).paragraphs[3], "Last wins and gamma two"); + assert.deepEqual(Object.keys(await entryBytes(out2)).sort(), beforeNames); +}); + +/* ---------------- diffComments ---------------- */ + +test("diffComments reports only ids that are new", () => { + const prev: Comment[] = [{ id: "1", author: "A", text: "old" }]; + const next: Comment[] = [ + { id: "1", author: "A", text: "changed wording" }, // same id, changed text: update, NOT added + { id: "2", author: "B", text: "brand new" }, // added + ]; + assert.deepEqual(diffComments(prev, next).added, [{ id: "2", author: "B", text: "brand new" }]); + assert.deepEqual(diffComments(next, prev).added, []); + assert.deepEqual(diffComments([], []).added, []); +}); + +/* ---------------- CRC validation on read ---------------- */ + +test("a zip entry whose data no longer matches its CRC-32 is rejected loudly (invalid_zip)", async () => { + // Build the docx with STORED entries so corrupting a payload byte breaks ONLY the CRC (deflate corruption + // could fail earlier in the inflate step and would not prove the CRC check itself). + const zip = new JSZip(); + zip.file("[Content_Types].xml", CONTENT_TYPES); + zip.file("_rels/.rels", ROOT_RELS); + zip.file("word/document.xml", DOCUMENT_XML); + const clean = await zip.generateAsync({ type: "nodebuffer", compression: "STORE" }); + assert.equal(parseDocx(clean).paragraphs[0], "Hello & welcome to <Office>.", "the uncorrupt STORED archive parses"); + + // Find the LFH of word/document.xml and flip one byte of its (stored) payload. + const needle = Buffer.from("word/document.xml", "utf8"); + const corrupted = Buffer.from(clean); + let flipped = false; + for (let i = 0; i + 30 + needle.length <= corrupted.length; i++) { + if (corrupted.readUInt32LE(i) !== 0x04034b50) continue; // local file header signature + const nameLen = corrupted.readUInt16LE(i + 26); + const extraLen = corrupted.readUInt16LE(i + 28); + if (nameLen !== needle.length || !corrupted.subarray(i + 30, i + 30 + nameLen).equals(needle)) continue; + corrupted[i + 30 + nameLen + extraLen] ^= 0xff; // stored: payload starts right after the name + flipped = true; + break; + } + assert.ok(flipped, "located the stored word/document.xml entry to corrupt"); + + const err = thrown(() => parseDocx(corrupted)); + assert.equal(err.code, "invalid_zip"); + assert.match(err.message, /CRC mismatch for word\/document\.xml/); + // and rewriting (persisting) the corrupt archive fails too -- corruption is never re-persisted with its old CRC + const rewriteErr = thrown(() => rewriteZip(corrupted, {})); + assert.equal(rewriteErr.code, "invalid_zip"); +}); + +/* ---------------- readZip archive-level expansion cap ---------------- */ + +test("readZip refuses an archive whose COMBINED advertised expansion exceeds the 1 GiB archive cap", () => { + // Hand-crafted zip with NO payload bytes: 6 central-directory records each advertising 200 MiB (deflate method) -- + // every entry stays UNDER the 256 MiB per-entry cap on purpose; 1.2 GiB total crosses the 1 GiB archive cap. + // readZip trips in its CD loop before any payload is read, so the buffer never needs real data. + const N = 6; + const rec = 46 + 8; // central-directory record (fixed part) + name "part.xml" + const cdSize = N * rec; + const buf = Buffer.alloc(cdSize + 22); + for (let i = 0; i < N; i++) { + const p = i * rec; + buf.writeUInt32LE(0x02014b50, p); // central directory file header signature + buf.writeUInt16LE(8, p + 10); // compression method (deflate) + buf.writeUInt32LE(200 * 1024 * 1024, p + 24); // uncompressed size -- under the per-entry cap ON PURPOSE + buf.writeUInt16LE(8, p + 28); // file name length + buf.write("part.xml", p + 46); + } + const eocd = cdSize; + buf.writeUInt32LE(0x06054b50, eocd); // end of central directory signature + buf.writeUInt16LE(N, eocd + 10); // total number of entries + buf.writeUInt32LE(cdSize, eocd + 12); // size of the central directory + buf.writeUInt32LE(0, eocd + 16); // offset of the central directory + assert.throws(() => parseDocx(buf), /archive cap/); +}); diff --git a/extensions/office/test/ping.test.mts b/extensions/office/test/ping.test.mts new file mode 100644 index 000000000..6699a35ad --- /dev/null +++ b/extensions/office/test/ping.test.mts @@ -0,0 +1,301 @@ +/** + * Unit tests for src/ping.ts (wake on comment mention, plan 5.4). Run from the repo ROOT: + * env -u NODE_ENV node --import tsx --test extensions/office/test/ping.test.mts + * Self-contained: tmp dataDir per test, real docx created via store.createDoc, comment XML injected + * with ooxml.rewriteZip, every side effect (folders/inFolder/tell/record/log) injected through + * wirePing so the tests NEVER reach portal(). + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { + wirePing, + __resetPingForTests, + handlePersisted, + drainWakeQueue, + officeWakeMessage, + type PingAuditEntry, + type WakeEntry, +} from "../src/ping"; +import { createDoc, getDoc, importFile, templateDocx } from "../src/store"; +import { rewriteZip, parseDocx } from "../src/ooxml"; + +/* ------------------------------------------------------------------ */ +/* fakes */ +/* ------------------------------------------------------------------ */ + +const FOLDERS = [ + { path: "/f/acme", name: "Acme", kind: "project" as const }, + { path: "/f/atlas", name: "Atlas", kind: "agent" as const }, + { path: "/f/helper", name: "Helper", kind: "agent" as const }, + { path: "/f/twin-agent", name: "Twin", kind: "agent" as const }, + { path: "/f/twin-project", name: "twin", kind: "project" as const }, // case-insensitive duplicate of the agent above -> ambiguous + { path: "/f/notes", name: "Notes", kind: "chat" as const }, // wrong kind: never a mention target +]; +const HOME_CHAT: Record<string, string> = { + "/f/acme": "chat-acme-home", + "/f/atlas": "chat-atlas-home", + "/f/helper": "chat-helper-home", + "/f/twin-agent": "twin-agent-chat", + "/f/twin-project": "twin-project-chat", +}; + +type Seen = { tells: [string, unknown][]; records: PingAuditEntry[]; logs: string[] }; + +function setup(): { root: string; seen: Seen } { + __resetPingForTests(); + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-ping-")); + const seen: Seen = { tells: [], records: [], logs: [] }; + wirePing({ + root, + folders: () => FOLDERS, + inFolder: (p) => (HOME_CHAT[p] ? [{ id: HOME_CHAT[p] }] : []), + tell: (chatId: string, data: unknown) => { + seen.tells.push([chatId, data]); + return true; + }, + record: (e) => seen.records.push(e), + log: (l) => seen.logs.push(l), + }); + return { root, seen }; +} + +async function makeDoc(root: string, title: string, opts?: { type?: "word" | "cell"; owner?: string | null }) { + const input = { title, type: opts?.type ?? ("word" as const) }; + if (opts && "owner" in opts) input.owner = opts.owner; + return await createDoc(input, root); +} + +const esc = (s: string) => s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">"); +type C = { id: string; author: string; text: string }; +function commentsXml(cs: C[]): string { + return ( + '<w:comments xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">' + + cs.map((c) => `<w:comment id="${c.id}" author="${esc(c.author)}" date="2026-01-01T00:00:00Z"><w:p><w:r><w:t xml:space="preserve">${esc(c.text)}</w:t></w:r></w:p></w:comment>`).join("") + + "</w:comments>" + ); +} + +/** Simulate a DS persist of the doc with exactly these comments in word/comments.xml. */ +function persistWith(root: string, docId: string, title: string, ext: string, cs: C[]): void { + const buf = fs.readFileSync(path.join(root, "docs", docId, `${title}.${ext}`)); + handlePersisted(docId, rewriteZip(buf, { "word/comments.xml": commentsXml(cs) })); +} + +const stateFile = (root: string, docId: string) => path.join(root, "state", `${docId}.json`); + +/* ------------------------------------------------------------------ */ +/* officeWakeMessage */ +/* ------------------------------------------------------------------ */ + +test("officeWakeMessage renders the exact plan 5.4 template", () => { + const msg = officeWakeMessage("Budget 2026", "Alice", "first line @agent\nsecond line"); + assert.equal( + msg, + [ + 'Alice asked you in a comment on the office document "Budget 2026":', + "> first line @agent", + "> second line", + "You have been given edit access to this document; answer with the office_* tools.", + ].join("\n") + ); + // single-line comment, no trailing blank line + assert.equal(officeWakeMessage("T", "Bob", "one"), 'Bob asked you in a comment on the office document "T":\n> one\nYou have been given edit access to this document; answer with the office_* tools.'); +}); + +/* ------------------------------------------------------------------ */ +/* persist behavior */ +/* ------------------------------------------------------------------ */ + +test("created doc, first save: a mention IS processed (no baseline swallows first-session intent)", async () => { + const { root, seen } = setup(); + const meta = await makeDoc(root, "Budget", { owner: "chat-atlas-home" }); + persistWith(root, meta.docId, meta.title, meta.ext, [ + { id: "1", author: "Alice", text: "@agent please review this" }, // created docs get NO baseline: the mention wakes on save #1 + { id: "2", author: "Bob", text: "notes for later" }, + ]); + assert.deepEqual(seen.tells, [["chat-atlas-home", { type: "ping" }]]); + assert.equal(seen.records.length, 1); + const state = JSON.parse(fs.readFileSync(stateFile(root, meta.docId), "utf8")) as { ids: string[] }; + assert.deepEqual(state.ids.sort(), ["1", "2"]); + const grants = getDoc(meta.docId, root)?.grants ?? []; + assert.equal(grants.length, 1); + assert.equal(grants[0].chatId, "chat-atlas-home"); + assert.equal(grants[0].commentId, "1"); +}); + +test("imported doc, first save: its existing comments are baselined WITHOUT wakes", async () => { + const { root, seen } = setup(); + // A file with comment history that predates this library -- the whole point of importFile(imported:true). + const src = path.join(root, "src-import.docx"); + fs.writeFileSync(src, rewriteZip(await templateDocx(), { "word/comments.xml": commentsXml([ + { id: "7", author: "Cara", text: "carry-over note" }, + { id: "8", author: "Dan", text: "old @agent mention" }, + ]) })); + const meta = importFile(src, { title: "Carried Over" }, root); // sync + assert.equal(meta.imported, true); + assert.deepEqual(meta.importedBaseline, ["7", "8"]); // stamped from the file at IMPORT time (review 4364810146) + + // A mention added AFTER the import but BEFORE the first save is NEW to this library (not in importedBaseline) -> it IS + // processed on save #1; the imported history (7, 8) baselines silently. + persistWith(root, meta.docId, meta.title, meta.ext, [ + { id: "7", author: "Cara", text: "carry-over note" }, + { id: "8", author: "Dan", text: "old @agent mention" }, + { id: "9", author: "Eli", text: "@project:acme please take over this doc" }, + ]); + assert.deepEqual(seen.tells, [["chat-acme-home", { type: "ping" }]]); // only the post-import mention wakes + assert.equal(seen.records.length, 1); + const state = JSON.parse(fs.readFileSync(stateFile(root, meta.docId), "utf8")) as { ids: string[] }; + assert.deepEqual(state.ids.sort(), ["7", "8", "9"]); + const grants = getDoc(meta.docId, root)?.grants ?? []; + assert.equal(grants.length, 1); + assert.equal(grants[0].commentId, "9"); + + // ...and a plain re-send of the same set adds nothing (pure id diff) + persistWith(root, meta.docId, meta.title, meta.ext, [ + { id: "7", author: "Cara", text: "carry-over note" }, + { id: "8", author: "Dan", text: "old @agent mention" }, + { id: "9", author: "Eli", text: "@project:acme please take over this doc" }, + ]); + assert.deepEqual(seen.tells, [["chat-acme-home", { type: "ping" }]]); // still just the one wake from save #1 +}); + +test("second persist: @agent wakes the owner; @project:<name> matches case-insensitively, grant + wake queued; re-send dedupes", async () => { + const { root, seen } = setup(); + const meta = await makeDoc(root, "Budget", { owner: "chat-atlas-home" }); + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "1", author: "Alice", text: "baseline" }]); + assert.deepEqual(seen.tells, []); + + const c2: C = { id: "2", author: "Alice", text: "@agent please review this" }; + const c3: C = { id: "3", author: "Bob", text: "hand off to @project:acme when ready" }; + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "1", author: "Alice", text: "baseline" }, c2, c3]); + + // owner wake with the EXACT template text, and project home resolved per plan 2.1 via inFolder(folder.path) + assert.deepEqual(seen.tells, [ + ["chat-atlas-home", { type: "ping" }], + ["chat-acme-home", { type: "ping" }], + ]); + const ownerQueue = drainWakeQueue("chat-atlas-home"); + assert.equal(ownerQueue.length, 1); + assert.deepEqual(ownerQueue[0], { text: officeWakeMessage(meta.title, c2.author, c2.text), details: { docId: meta.docId, commentId: "2" } }); + const acmeQueue = drainWakeQueue("chat-acme-home") as WakeEntry[]; + assert.equal(acmeQueue.length, 1); + assert.deepEqual(acmeQueue[0].text, officeWakeMessage(meta.title, c3.author, c3.text)); + assert.deepEqual(acmeQueue[0].details, { docId: meta.docId, commentId: "3" }); + + // grants added (deduped by commentId) and audited + const grants = getDoc(meta.docId, root)?.grants ?? []; + for (const g of grants) assert.equal(typeof g.at, "number"); + assert.ok(grants.some((g) => g.chatId === "chat-atlas-home" && g.commentId === "2")); + assert.ok(grants.some((g) => g.chatId === "chat-acme-home" && g.commentId === "3")); + assert.equal(grants.length, 2); + assert.equal(seen.records.length, 2); + for (const r of seen.records) { + assert.equal(r.subject, meta.title); + assert.match(r.reason, /comment mention/); + assert.ok(["chat-atlas-home", "chat-acme-home"].includes(r.chatId)); + } + + // re-send the SAME comments: no second ping, no second grant + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "1", author: "Alice", text: "baseline" }, c2, c3]); + assert.deepEqual(seen.tells.length, 2); + assert.equal(getDoc(meta.docId, root)?.grants?.length, 2); + assert.equal(drainWakeQueue("chat-acme-home").length, 0); + + // state now holds all seen ids + const state = JSON.parse(fs.readFileSync(stateFile(root, meta.docId), "utf8")) as { ids: string[] }; + assert.deepEqual(state.ids.sort(), ["1", "2", "3"]); +}); + +test("bare @name matches an agent folder case-insensitively; unmatched and ambiguous names log only; duplicate mention of one target pings once", async () => { + const { root, seen } = setup(); + const meta = await makeDoc(root, "Plan", { owner: null }); + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "1", author: "A", text: "baseline" }]); + + persistWith(root, meta.docId, meta.title, meta.ext, [ + { id: "1", author: "A", text: "baseline" }, + { id: "2", author: "B", text: "cc @HELPER on the draft" }, // folder is named "Helper" + { id: "3", author: "C", text: "whoever reads this, @nobody help" }, // no such folder + { id: "4", author: "D", text: "@twin please sort this out" }, // agent + project both named Twin (case-insensitive) -> ambiguous + { id: "5", author: "E", text: "@acme and again @ACME, one job" }, // same target twice in one comment + ]); + + assert.deepEqual(seen.tells, [ + ["chat-helper-home", { type: "ping" }], + ["chat-acme-home", { type: "ping" }], + ]); + const logs = seen.logs.join("\n"); + assert.match(logs, /@nobody .* matches no agent or project/); + assert.match(logs, /@twin is ambiguous .*(agent|project).*Twin.*-- no silent disambiguation/); + // chat-kind folders are never mention targets: @notes would be unmatched (not asserted above) -- but grants confirm only real targets landed + const grants = getDoc(meta.docId, root)?.grants ?? []; + assert.deepEqual( + grants.map((g) => [g.chatId, g.commentId]).sort(), + [["chat-acme-home", "5"], ["chat-helper-home", "2"]].sort() + ); +}); + +test("@agent on an ownerless doc is skipped with a log line, no crash", async () => { + const { root, seen } = setup(); + const meta = await makeDoc(root, "Solo", {}); // owner omitted -> null + assert.equal(meta.owner, null); + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "1", author: "A", text: "baseline" }]); + persistWith(root, meta.docId, meta.title, meta.ext, [ + { id: "1", author: "A", text: "baseline" }, + { id: "2", author: "B", text: "@agent fix the totals" }, + ]); + assert.deepEqual(seen.tells, []); + assert.deepEqual(seen.records, []); + assert.match(seen.logs.join("\n"), /@agent .* ownerless document -- no wake/); + // ids still recorded so a later save diff stays sane + const state = JSON.parse(fs.readFileSync(stateFile(root, meta.docId), "utf8")) as { ids: string[] }; + assert.deepEqual(state.ids.sort(), ["1", "2"]); +}); + +test("xlsx persist is a no-op (v1 wakes docx only)", async () => { + const { root, seen } = setup(); + const meta = await makeDoc(root, "Sheet", { type: "cell", owner: "chat-atlas-home" }); + assert.equal(meta.ext, "xlsx"); + // even a complete garbage buffer must be ignored without touching state or pings + handlePersisted(meta.docId, Buffer.from("not a zip at all")); + assert.deepEqual(seen.tells, []); + assert.ok(!fs.existsSync(stateFile(root, meta.docId))); + assert.equal(getDoc(meta.docId, root)?.grants?.length ?? 0, 0); +}); + +test("corrupt state file: logged, previous set treated as empty, mentions processed once, state restored", async () => { + const { root, seen } = setup(); + const meta = await makeDoc(root, "Ledger", { owner: "chat-atlas-home" }); + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "1", author: "A", text: "@agent baseline ask" }]); // created doc: the @agent mention IS processed on its very first save (no baseline) + fs.writeFileSync(stateFile(root, meta.docId), "{ this is not json"); + + persistWith(root, meta.docId, meta.title, meta.ext, [ + { id: "1", author: "A", text: "@agent baseline ask" }, // corrupt state -> empty seen set: "new" again -- re-wake accepted (see module docs) + { id: "2", author: "B", text: "@agent second ask" }, + ]); + assert.match(seen.logs.join("\n"), /state file .* corrupt/); + // one ping from the first save, then BOTH comments re-process after the corruption (queue dedupes per comment, but the tell fires each pass) + assert.deepEqual(seen.tells, [ + ["chat-atlas-home", { type: "ping" }], + ["chat-atlas-home", { type: "ping" }], + ["chat-atlas-home", { type: "ping" }], + ]); + // wake queue dedupes per (docId, commentId) but grants also dedupe: both asks land once + const grants = getDoc(meta.docId, root)?.grants ?? []; + assert.deepEqual( + grants.map((g) => [g.chatId, g.commentId]).sort(), + [["chat-atlas-home", "1"], ["chat-atlas-home", "2"]].sort() + ); + const state = JSON.parse(fs.readFileSync(stateFile(root, meta.docId), "utf8")) as { ids: string[] }; + assert.deepEqual(state.ids.sort(), ["1", "2"]); // restored to a sane file +}); + +test("sanity: the injected comments are really parsed from the rewritten docx (fixture self-check)", async () => { + const { root } = setup(); + const meta = await makeDoc(root, "Selfcheck", {}); + persistWith(root, meta.docId, meta.title, meta.ext, [{ id: "9", author: "Zoe", text: "multi\nword check @agent" }]); + const buf = fs.readFileSync(path.join(root, "docs", meta.docId, `${meta.title}.docx`)); + assert.equal(parseDocx(rewriteZip(buf, { "word/comments.xml": commentsXml([{ id: "9", author: "Zoe", text: "multi\nword check @agent" }]) })).comments.length, 1); +}); diff --git a/extensions/office/test/server.test.mts b/extensions/office/test/server.test.mts new file mode 100644 index 000000000..a5dabe169 --- /dev/null +++ b/extensions/office/test/server.test.mts @@ -0,0 +1,862 @@ +/** + * Unit tests for src/server.ts (T-WIRE): the boot wiring, the loopback listener, and every UI route -- against a + * FAKE portal (mini router + recorded log/audit/extensions.use) with real tmp dataDirs and a recording fake Docker + * literal. Nothing here reaches the real portal(), the Docker socket, or any network host but 127.0.0.1 loopback + * (the listener itself). Booted instances are cleaned up at the end of each test; live-module presence is reset for + * isolation because it is process-global. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import http from "node:http"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import zlib from "node:zlib"; +import type { AddressInfo } from "node:net"; +import type { ServerPortal } from "@pithagoras/extension-api/server"; +import type { ContainerInfo, Docker, ExecResult, VolumeInfo } from "../../docker/src/api"; +import officeServer, { dsOrigin, jwtSecret, loopbackPort, settleKept } from "../src/server"; +import * as store from "../src/store"; +import { rewriteZip } from "../src/ooxml"; +import * as live from "../src/live"; +import { mintTicket, verifyTicket } from "../src/tickets"; +import { verifyJwt } from "../src/jwt"; +import { makeKey, parseKey, revHashOf } from "../src/callback"; + +const DEFAULT_TAG = "onlyoffice/documentserver:10.0"; + +/* ------------------------------------------------------------------ */ +/* fakes */ +/* ------------------------------------------------------------------ */ + +class MiniRouter { + routes: Array<{ method: string; pattern: string | RegExp; handler: (req: unknown, res: FakeRes) => Promise<void> | void }> = []; + private reg(method: string) { + return (pattern: string | RegExp, ...rest: unknown[]) => { + const handler = rest[rest.length - 1] as (req: unknown, res: FakeRes) => Promise<void> | void; + this.routes.push({ method, pattern, handler }); + }; + } + /** Express-shaped registration: (pattern, ...middlewares, handler). */ + get(...args: unknown[]) { this.reg("GET")(...args); } + post(...args: unknown[]) { this.reg("POST")(...args); } + put(...args: unknown[]) { this.reg("PUT")(...args); } + delete(...args: unknown[]) { this.reg("DELETE")(...args); } + + /** Express-shaped fake request: the plain fields PLUS the stream surface -- an async iterator over the body + * (real IncomingMessages are async-iterable, and the /ds proxy's POST forwarding reads `for await (req)`). */ + private fakeReq(method: string, urlPath: string, params: Record<string, string>, body: unknown, headers: Record<string, string>): unknown { + const o: Record<PropertyKey, unknown> = { method, url: urlPath, params, query: {}, body, headers }; + const data = typeof body === "string" ? Buffer.from(body, "utf8") : Buffer.isBuffer(body) ? body : undefined; + o[Symbol.asyncIterator] = async function* () { if (data && data.length > 0) yield data; }; + return o; + } + + async dispatch(method: string, urlPath: string, body: unknown, headers: Record<string, string> = {}): Promise<FakeRes> { + const us = urlPath.split("/").filter(Boolean); + for (const route of this.routes) { + if (route.method !== method) continue; + // Express regex paths match the WHOLE raw url (query included); groups land in params[0..]. + if (typeof route.pattern === "object") { + const m = route.pattern.exec(urlPath); + if (!m) continue; + const params: Record<string, string> = {}; + for (let g = 1; g < m.length; g++) params[String(g - 1)] = m[g] ?? ""; + const res = fakeRes(); + await route.handler(this.fakeReq(method, urlPath, params, body, headers), res); + return res; + } + const ps = route.pattern.split("/").filter(Boolean); + if (ps.length !== us.length) continue; + const params: Record<string, string> = {}; + let hit = true; + for (let i = 0; i < ps.length; i++) { + if (ps[i].startsWith(":")) params[ps[i].slice(1)] = decodeURIComponent(us[i]); + else if (ps[i] !== us[i]) { hit = false; break; } + } + if (!hit) continue; + const res = fakeRes(); + await route.handler(this.fakeReq(method, urlPath, params, body, headers), res); + return res; + } + throw new Error(`no registered ${method} route for ${urlPath}`); + } +} + +function fakeRes(): FakeRes { + let code = 200; + const r = { + headers: {} as Record<string, unknown>, + data: undefined as unknown, + body: undefined as unknown, + status(c: number) { code = c; return r; }, + json(o: unknown) { r.body = o; return r; }, + setHeader(k: string, v: unknown) { r.headers[k.toLowerCase()] = v; }, + writeHead(c: number, h?: Record<string, string>) { code = c; if (h) for (const k of Object.keys(h)) r.headers[k.toLowerCase()] = h[k]; }, + end(b?: unknown) { r.data = b; }, + }; + Object.defineProperty(r, "statusCode", { get: () => code }); + return r as unknown as FakeRes; +} + +type FakeRes = { + headers: Record<string, unknown>; + data: unknown; + body: unknown; + statusCode: number; +}; + +function makeDocker(opts: { networkMode?: string } = {}) { + const state = { container: undefined as ContainerInfo | undefined, volumes: [] as VolumeInfo[], inspectBody: undefined as Record<string, unknown> | undefined, execResult: { output: "", exitCode: 1, timedOut: false } as ExecResult, networkMode: opts.networkMode ?? "host" }; + const calls = { ensure: [] as string[], create: [] as Array<[string, Record<string, unknown>]>, start: [] as string[], stop: [] as string[] }; + const docker = { + available: async () => true, + sharedNetwork: async () => state.networkMode, + images: { present: async () => true, ensure: async (image: string) => { calls.ensure.push(image); } }, + containers: { + list: async () => (state.container ? [state.container] : []), + get: async () => state.container, + create: async (name: string, spec: Record<string, unknown>) => { + calls.create.push([name, spec]); + const info: ContainerInfo = { name, id: `cid-${calls.create.length}`, image: String(spec.Image), state: "created", status: "Created", owner: "", labels: {} }; + state.container = info; + return info; + }, + start: async (name: string) => { calls.start.push(name); if (state.container && name === state.container.name) state.container.state = "running"; }, + stop: async (name: string) => { calls.stop.push(name); if (state.container && name === state.container.name) state.container.state = "exited"; }, + remove: async () => {}, logs: async () => "", inspect: async () => state.inspectBody, + exec: async () => state.execResult, adopt: async () => { throw Object.assign(new Error("nope"), { status: 404 }); }, + }, + volumes: { list: async () => [...state.volumes], create: async (n: string) => { state.volumes.push({ name: n, owner: "" }); }, remove: async () => {}, adopt: async () => { throw Object.assign(new Error("nope"), { status: 404 }); } }, + } as unknown as Docker; + return { docker, calls, state }; +} + +interface Booted { + root: string; + portal: ServerPortal; + router: MiniRouter; + logs: string[]; + audits: Array<Record<string, unknown>>; + allowedDirs: string[]; + cleanup: () => Promise<void>; +} + +async function boot(root?: string, docker?: Docker): Promise<Booted> { + const rootDir = root ?? fs.mkdtempSync(path.join(os.tmpdir(), "office-server-")); + const logs: string[] = []; + const audits: Array<Record<string, unknown>> = []; + const allowedDirs: string[] = []; + const router = new MiniRouter(); + const ddock = docker ?? makeDocker().docker; // ONE instance per boot: extensions.use() resolves the same fake every time + const portalObj = { + extension: "pithagoras-office", + dataDir: () => rootDir, + log: (l: unknown) => logs.push(String(l)), + audit: { record: (e: Record<string, unknown>) => audits.push(e) }, + router, + http: { json: (_o?: unknown) => (_req: unknown, _res: unknown, next: () => void) => next(), text: () => () => undefined, raw: () => () => undefined, send: () => () => {}, upgrade: () => undefined }, + folders: { list: () => [] as never[], allowed: (p: string) => allowedDirs.some((d) => p === d || p.startsWith(d + path.sep)), label: () => ({ name: "" }) }, + chats: { extend: (_f: unknown) => {}, get: () => undefined, list: () => [], inFolder: () => [] as never[], folder: () => { throw new Error("none"); }, send: () => {}, tell: async () => false }, + extensions: { use: () => ddock }, + } as unknown as ServerPortal; + const cleanup = await officeServer(portalObj); + return { root: rootDir, portal: portalObj, router, logs, audits, allowedDirs, cleanup }; +} + +async function until(cond: () => boolean, ms = 2000): Promise<void> { + const t0 = Date.now(); + while (!cond()) { + if (Date.now() - t0 > ms) throw new Error("timed out waiting"); + await new Promise((r) => setTimeout(r, 10)); + } +} + +function writeDsJson(root: string, origin = "http://127.0.0.1"): void { + fs.writeFileSync(path.join(root, "ds.json"), JSON.stringify({ containerId: "cid-test", origin, imageTag: DEFAULT_TAG })); +} + +/* ---------------- boot: jwt-secret + cleanup ---------------- */ + +test("boot creates jwt-secret 0600 exactly once; a second boot reuses the bytes verbatim", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-secret-")); + const b1 = await boot(root); + const file = path.join(root, "jwt-secret"); + assert.ok(fs.existsSync(file), "jwt-secret created at boot"); + assert.equal(fs.statSync(file).mode & 0o777, 0o600, "mode is exactly 0600 (umask-proof)"); + const s1 = jwtSecret(root); + assert.match(s1, /^[0-9a-f]{64}$/, "trimmed content is a 64-hex-char random value"); + + const b2 = await boot(root); // same dataDir, second portal start + assert.equal(jwtSecret(root), s1, "second boot reuses the existing secret byte-for-byte"); + const before = fs.readFileSync(file, "utf8"); + const b3 = await boot(root); + assert.equal(fs.readFileSync(file, "utf8"), before); + + await b3.cleanup(); await b2.cleanup(); await b1.cleanup(); +}); + +test("cleanup closes the loopback listener and clears the timer (nothing keeps node alive)", async () => { + const b = await boot(); + const port = loopbackPort(); + assert.ok(port > 0, "the listener bound a real ephemeral loopback port"); + let ok = false; + try { + const r = await fetch(`http://127.0.0.1:${port}/nope`); + ok = r.status === 404; // reachable BEFORE cleanup, unknown path -> 404 JSON + } catch { assert.fail("loopback was not reachable before cleanup"); } + assert.ok(ok); + + await b.cleanup(); + assert.equal(loopbackPort(), 0, "port state cleared after cleanup"); + let refused = false; + try { + await fetch(`http://127.0.0.1:${port}/nope`); + } catch { + refused = true; + } + assert.ok(refused, "the listener is closed: connections are refused after cleanup"); +}); + +/* ---------------- loopback listener ---------------- */ + +test("loopback GET /doc/<ticket> returns the exact bytes + content-type + no-store; bad/expired tickets -> 401", async () => { + const b = await boot(); + const doc = await store.createDoc({ title: "Loop", type: "word" }, b.root); + const secret = jwtSecret(b.root); + const { token } = mintTicket(doc.docId, 600_000, secret); + + const r = await fetch(`http://127.0.0.1:${loopbackPort()}/doc/${token}`); + assert.equal(r.status, 200); + assert.equal(r.headers.get("content-type"), "application/vnd.openxmlformats-officedocument.wordprocessingml.document"); + assert.equal(r.headers.get("cache-control"), "no-store"); + const disk = fs.readFileSync(path.join(b.root, "docs", doc.docId, `${doc.title}.docx`)); + assert.ok(Buffer.compare(Buffer.from(await r.arrayBuffer()), disk) === 0, "served bytes are exactly the library bytes"); + + // xlsx gets the spreadsheet media type + const cell = await store.createDoc({ title: "Cells", type: "cell" }, b.root); + const { token: t2 } = mintTicket(cell.docId, 600_000, secret); + const r2 = await fetch(`http://127.0.0.1:${loopbackPort()}/doc/${t2}`); + assert.equal(r2.headers.get("content-type"), "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"); + + // invalid ticket -> 401 JSON; expired ticket -> 401 JSON (negative ttl mints an already-expired one) + const r3 = await fetch(`http://127.0.0.1:${loopbackPort()}/doc/garbage.ticket.here`); + assert.equal(r3.status, 401); + assert.deepEqual(await r3.json(), { error: "invalid or expired document ticket", code: "bad_ticket" }); + const { token: expired } = mintTicket(doc.docId, -1_000, secret); + assert.equal(verifyTicket(expired, secret), null, "sanity: the negative-ttl ticket does not verify"); + const r4 = await fetch(`http://127.0.0.1:${loopbackPort()}/doc/${expired}`); + assert.equal(r4.status, 401); + + live.clearDoc(doc.docId); live.clearDoc(cell.docId); + await b.cleanup(); +}); + +test("loopback POST /callback while the DS is unconfigured -> 503 {error:1}; unknown paths -> 404 JSON", async () => { + const b = await boot(); // fresh root: no ds.json yet, but configure() ran at boot with this root's lookups + const r = await fetch(`http://127.0.0.1:${loopbackPort()}/callback`, { method: "POST", headers: { "content-type": "application/json" }, body: "{}" }); + assert.equal(r.status, 503); + assert.deepEqual(await r.json(), { error: 1 }); // the DS callback convention, not our JSON shape + + const r2 = await fetch(`http://127.0.0.1:${loopbackPort()}/whatever`); + assert.equal(r2.status, 404); + const body = (await r2.json()) as Record<string, unknown>; + assert.equal(typeof body.error, "string"); + assert.equal(body.code, "not_found"); + + const r3 = await fetch(`http://127.0.0.1:${loopbackPort()}/callback`); // GET on the POST-only route is also 404 + assert.equal(r3.status, 404); + await b.cleanup(); +}); + +/* ---------------- /editor-config ---------------- */ + +test("editor-config (docx): key round-trips through parseKey, url is a fresh verifiable loopback ticket, the whole config is JWT-signed", async () => { + const b = await boot(); + writeDsJson(b.root); + const doc = await store.createDoc({ title: "Plan", type: "word" }, b.root); + const secret = jwtSecret(b.root); + + const res = await b.router.dispatch("POST", "/editor-config", { doc: doc.docId }); + assert.equal(res.statusCode, 200); + const body = res.body as Record<string, unknown> & { token: string; apiJs: string }; + + // the plain fields a DocsAPI.DocEditor gets + assert.equal(body.documentType, "word"); + assert.equal(body.type, "embedded"); + assert.equal(body.title, "Plan"); + const document = body.document as Record<string, unknown>; + assert.equal(document.fileType, "docx"); + assert.equal(document.title, "Plan.docx"); + // key: exactly makeKey(docId, current mtime) and parseable back + assert.equal(document.key, makeKey(doc.docId, doc.mtimeMs)); + const parsed = parseKey(String(document.key)) as { docId: string; revHash: string }; + assert.equal(parsed.docId, doc.docId); + assert.equal(parsed.revHash, revHashOf(doc.mtimeMs)); + // url: on OUR loopback listener (plan 5.2), ticketed, and the ticket verifies with the boot secret + const url = String(document.url); + assert.ok(url.startsWith(`http://127.0.0.1:${loopbackPort()}/doc/`), `url uses the loopback origin: ${url}`); + const ticket = url.split("/doc/")[1]; + assert.ok(ticket.length > 0 && !ticket.includes("/")); + assert.equal(verifyTicket(ticket, secret)?.docId, doc.docId, "the minted ticket verifies with the boot secret"); + // editorConfig per plan 5.6 + const ec = body.editorConfig as Record<string, unknown>; + assert.equal(ec.callbackUrl, `http://127.0.0.1:${loopbackPort()}/callback`); + assert.equal(ec.mode, "edit"); + assert.deepEqual(ec.user, { id: "portal-user", name: "Portal" }); + // the JWT wrap: payload == the WHOLE plain config (minus token/apiJs), 15-minute life, verifies with the same secret + const claims = verifyJwt(body.token, secret); + assert.ok(claims, "the signed config verifies"); + assert.deepEqual( + { documentType: claims?.documentType, type: claims?.type, title: claims?.title, document: claims?.document, editorConfig: claims?.editorConfig }, + { documentType: body.documentType, type: body.type, title: body.title, document: body.document, editorConfig: body.editorConfig }, + ); + const exp = Number(claims?.exp); const iat = Number(claims?.iat); + assert.equal(exp - iat, 900, "config token lives exactly 15 minutes (the official sample's window)"); + // api.js src: ROOT-RELATIVE through this extension's own /ds proxy (resolved by the browser against the page + // origin it reached the portal at); getBasePath() turns that prefix into the editor iframe base. The loopback + // document.url/callbackUrl above are what STAY absolute for DS-side fetches. + assert.equal(body.apiJs, "/api/x/pithagoras-office/ds/web-apps/apps/api/documents/api.js"); + // and minting claimed the seat in live presence + assert.deepEqual(live.usersOf(doc.docId).map((u) => ({ id: u.id, name: u.name })), [{ id: "portal-user", name: "Portal" }]); + + live.clearDoc(doc.docId); + await b.cleanup(); +}); + +test("editor-config (xlsx): documentType cell; a container:<id> origin refuses with the M0 error", async () => { + const b = await boot(); + writeDsJson(b.root); + const cell = await store.createDoc({ title: "Budget", type: "cell" }, b.root); + let res = await b.router.dispatch("POST", "/editor-config", { doc: cell.docId }); + assert.equal(res.statusCode, 200); + assert.equal((res.body as Record<string, unknown>).documentType, "cell"); + live.clearDoc(cell.docId); + + // the deployment shape with no browser-reachable URL in v1: dsOrigin returns the marker AS-IS and /editor-config refuses + writeDsJson(b.root, "container:9f3a1c2d"); + assert.equal(dsOrigin(b.root), "container:9f3a1c2d"); + res = await b.router.dispatch("POST", "/editor-config", { doc: cell.docId }); + assert.equal(res.statusCode, 422); + const body = res.body as Record<string, unknown>; + assert.equal(body.code, "unresolvable_ds_origin"); + assert.match(String(body.error), /M0/); + + // and with no ds.json at all: 503 not-configured; unknown doc: 404 + fs.rmSync(path.join(b.root, "ds.json")); + res = await b.router.dispatch("POST", "/editor-config", { doc: cell.docId }); + assert.equal(res.statusCode, 503); + res = await b.router.dispatch("POST", "/editor-config", { doc: "nope" }); + assert.equal(res.statusCode, 404); + assert.equal((res.body as Record<string, unknown>).code, "not_found"); + + writeDsJson(b.root); // leave a sane record for anyone reusing this root later + await b.cleanup(); +}); + +/* ---------------- /state ---------------- */ + +test("/state: settings + container status + docs list + locks for EVERY doc", async () => { + const b = await boot(); + const d1 = await store.createDoc({ title: "Alpha", type: "word" }, b.root); + const d2 = await store.createDoc({ title: "Beta", type: "cell" }, b.root); + live.setUser(d1.docId, { id: "u-ada", name: "Ada" }, true); + + const res = await b.router.dispatch("GET", "/state"); + assert.equal(res.statusCode, 200); + const body = res.body as Record<string, unknown>; + assert.deepEqual(Object.keys(body).sort(), ["container", "docs", "enabled", "imageTag", "locks"]); + assert.equal(body.enabled, false); // defaults from a fresh dataDir + assert.equal(body.imageTag, DEFAULT_TAG); + const c = body.container as Record<string, unknown>; + assert.equal(c.state, "stopped"); // fake docker: nothing made yet + const docs = body.docs as store.DocMeta[]; + assert.equal(docs.length, 2); + assert.deepEqual(docs.map((d) => d.title).sort(), ["Alpha", "Beta"]); + const locks = body.locks as Record<string, { id: string; name?: string }[]>; + assert.deepEqual(locks[d1.docId], [{ id: "u-ada", name: "Ada" }]); + assert.deepEqual(locks[d2.docId], [], "every doc gets a locks entry -- empty when nobody is in it"); + + live.resetLive(); + await b.cleanup(); +}); + +/* ---------------- /documents CRUD ---------------- */ + +test("/documents: create (409 on duplicate), rename keeps mtimeMs, download bytes+disposition, delete clears live presence", async () => { + const b = await boot(); + + let res = await b.router.dispatch("POST", "/documents", { title: "Plan", type: "word" }); + assert.equal(res.statusCode, 201); + const meta = res.body as store.DocMeta; + assert.equal(meta.ext, "docx"); + assert.ok(meta.docId.length > 0); + + res = await b.router.dispatch("POST", "/documents", { title: "Plan", type: "word" }); // same title+ext -> exists + assert.equal(res.statusCode, 409); + assert.equal((res.body as Record<string, unknown>).code, "exists"); + + res = await b.router.dispatch("POST", "/documents", { title: "", type: "word" }); + assert.equal(res.statusCode, 400); + res = await b.router.dispatch("POST", "/documents", { title: "X", type: "slides" }); + assert.equal(res.statusCode, 400); + + // rename: a new title, the SAME mtimeMs (open sessions' revision survives a retitle) + const before = store.getDoc(meta.docId, b.root) as store.DocMeta; + res = await b.router.dispatch("POST", `/documents/${meta.docId}/rename`, { title: "Plan v2" }); + assert.equal(res.statusCode, 200); + const after = res.body as store.DocMeta; + assert.equal(after.title, "Plan v2"); + assert.equal(after.mtimeMs, before.mtimeMs); + + // download: the exact disk bytes with an attachment disposition + live.setUser(meta.docId, { id: "u-1", name: "U" }, true); // set presence BEFORE delete to prove clearDoc + res = await b.router.dispatch("GET", `/documents/${meta.docId}`); + assert.equal(res.statusCode, 200); + const disk = fs.readFileSync(path.join(b.root, "docs", meta.docId, "Plan v2.docx")); + assert.ok(Buffer.compare(Buffer.from(res.data as Buffer), disk) === 0); + assert.equal(res.headers["content-type"], "application/vnd.openxmlformats-officedocument.wordprocessingml.document"); + assert.match(String(res.headers["content-disposition"]), /attachment; filename="Plan v2\.docx"/); + + // delete: gone, and the live doc cleared with it + res = await b.router.dispatch("DELETE", `/documents/${meta.docId}`); + assert.equal(res.statusCode, 200); + assert.deepEqual(live.usersOf(meta.docId), []); + res = await b.router.dispatch("GET", `/documents/${meta.docId}`); + assert.equal(res.statusCode, 404); + // export of a gone doc refuses too + res = await b.router.dispatch("GET", `/documents/${meta.docId}/export`); + assert.equal(res.statusCode, 404); + + live.resetLive(); + await b.cleanup(); +}); + +/* ---------------- /documents/rescan (the import route is gone) ---------------- */ + +/** Point the fake portal's folders.list at a workspace AFTER boot: the rescan route reads it live per request, + * so overwriting the method is enough -- exactly how a real portal picks up newly added agent folders. */ +function pointFoldersAt(b: Booted, ...dirs: string[]): void { + // the route reads AgentFolder[] ({path,name,kind}) -- the fake must hand back that shape, not bare strings + (b.portal as unknown as { folders: { list: () => Array<{ path: string; name: string; kind: string }> } }).folders.list = + () => dirs.map((d) => ({ path: d, name: path.basename(d), kind: "project" })); +} + +test("/documents/rescan: indexes in place (no copy), skips hidden/node_modules/.txt, second run is a no-op", async () => { + const b = await boot(); // boots with folders.list -> [] : the boot scan logs zero and touches nothing + assert.ok(b.logs.some((l) => l.startsWith("workspace scan at boot:")), "boot logged its workspace scan"); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-ws-")); + const srcBytes = Buffer.from(`a docx of sorts ${Date.now()}`); + fs.writeFileSync(path.join(ws, "report.docx"), srcBytes); + fs.mkdirSync(path.join(ws, "deep", "drafts"), { recursive: true }); // nested dirs ARE walked + fs.writeFileSync(path.join(ws, "deep", "drafts", "plan.xlsx"), Buffer.from("an xlsx of sorts")); + fs.mkdirSync(path.join(ws, ".hidden")); + fs.writeFileSync(path.join(ws, ".hidden", "secret.docx"), srcBytes); // hidden dir: never indexed +fs.mkdirSync(path.join(ws, "node_modules", "pkg"), { recursive: true }); + fs.writeFileSync(path.join(ws, "node_modules", "pkg", "bundled.docx"), srcBytes); // node_modules: never indexed + fs.writeFileSync(path.join(ws, "notes.txt"), "plain text is not office"); // wrong type: ignored, not an error + pointFoldersAt(b, ws); + + const res = await b.router.dispatch("POST", "/documents/rescan"); + assert.equal(res.statusCode, 200); + assert.deepEqual(res.body, { added: 2, updated: 0 }); + const docs = store.listDocs(b.root); + assert.equal(docs.length, 2); + const report = docs.find((d) => d.title === "report")!; + assert.equal(report.path, path.join(ws, "report.docx"), "the entry points at the REAL file"); + assert.ok(fs.existsSync(path.join(ws, "report.docx")), "the source file is still where it was: IN PLACE, never copied"); + assert.ok(!fs.existsSync(path.join(b.root, "docs", report.docId)), "nothing was written into the library's docs/ for a scanned doc"); + assert.equal(report.owner, null); // a human doc: every home's view shows it (docsForHome's unowned pass) + assert.equal(report.imported, true); + + // rescan again with nothing changed: idempotent by realpath identity, counts stay honest + const again = await b.router.dispatch("POST", "/documents/rescan"); + assert.deepEqual(again.body, { added: 0, updated: 0 }); + assert.equal(store.listDocs(b.root).length, 2); + + // something wrote the file straight to disk (an agent outside DS): a rescan REFRESHES it in place + const bigger = Buffer.concat([srcBytes, Buffer.from(" more")]); + fs.writeFileSync(path.join(ws, "report.docx"), bigger); + const refresh = await b.router.dispatch("POST", "/documents/rescan"); + assert.deepEqual(refresh.body, { added: 0, updated: 1 }); + const refreshed = store.listDocs(b.root).find((d) => d.docId === report.docId)!; + assert.equal(refreshed.sizeBytes, bigger.length); + assert.ok(Math.abs(refreshed.mtimeMs - fs.statSync(path.join(ws, "report.docx")).mtimeMs) < 50, "mtimeMs now matches the file on disk"); + + // /state's payload carries the sort keys: mtimeMs and sizeBytes are real numbers, path is where it lives + const state = (await b.router.dispatch("GET", "/state")).body as { docs: Array<Record<string, unknown>> }; + const stReport = state.docs.find((d) => d.docId === report.docId)!; + assert.equal(typeof stReport.mtimeMs, "number"); + assert.equal(stReport.sizeBytes, bigger.length); + assert.equal(stReport.path, path.join(ws, "report.docx")); + + // export of a scanned doc serves the REAL file's bytes as an attachment (fileFor honors DocMeta.path) + const ex = await b.router.dispatch("GET", `/documents/${report.docId}/export`); + assert.equal(ex.statusCode, 200); + assert.ok(Buffer.compare(Buffer.from(ex.data as Buffer), bigger) === 0); + assert.match(String(ex.headers["content-disposition"]), /filename="report\.docx"/); + + // the manual import surface is GONE: no route registered at all (MiniRouter throws, not a coded refusal) + await assert.rejects(b.router.dispatch("POST", "/import", { path: path.join(ws, "report.docx") }), /no registered POST route for \/import/); + + fs.rmSync(ws, { recursive: true, force: true }); + await b.cleanup(); +}); + +test("/documents/rescan: a docx with pre-existing comments is stamped with an importedBaseline at index time", async () => { + const b = await boot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-ws-baseline-")); + const docBytes = rewriteZip(await store.templateDocx(), { "word/comments.xml": commentsXmlWith([ + { id: "1", author: "Cara", text: "a mention from before the library existed" }, + ]) }); + fs.writeFileSync(path.join(ws, "legacy.docx"), docBytes); + pointFoldersAt(b, ws); + + const res = await b.router.dispatch("POST", "/documents/rescan"); + assert.equal(res.statusCode, 200); + const doc = store.listDocs(b.root).find((d) => d.title === "legacy")!; + assert.deepEqual(doc.importedBaseline, ["1"], "the file's own comment history is the first-save baseline -- index time must not wake anyone"); + fs.rmSync(ws, { recursive: true, force: true }); + await b.cleanup(); +}); + +/** Minimal comments.xml body for a docx (ping.test has its own richer variant). */ +function commentsXmlWith(comments: Array<{ id: string; author: string; text: string }>): string { + const items = comments.map((c) => `<w:comment w:id="${c.id}" w:author="${c.author}"><w:p><w:r><w:t>${c.text}</w:t></w:r></w:p></w:comment>`).join(""); + return `<?xml version="1.0" encoding="UTF-8" standalone="yes"?><w:comments xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">${items}</w:comments>`; +} + +/* ---------------- /settings + container routes ---------------- */ + +test("/settings: enable fires ensureContainer exactly once with the settings tag; disable stops + leaves the user-stopped marker", async () => { + const m = makeDocker(); + const b = await boot(undefined, m.docker); + assert.equal(store.listDocs(b.root).length, 0); + // a LEGACY settings file (written before the removed switch existed) must load without crashing -- write it + // before any POST so every route in this test runs over tolerant loading; saveSettings then rewrites it clean. + fs.writeFileSync(path.join(b.root, "settings.json"), JSON.stringify({ enabled: false, imageTag: DEFAULT_TAG, coEdit: true })); + + // enable: side effect is fire-and-forget -- poll for the create, not the response + let res = await b.router.dispatch("POST", "/settings", { enabled: true }); + assert.equal(res.statusCode, 200); + const body = res.body as Record<string, unknown>; + assert.equal((body.settings as Record<string, unknown>).enabled, true); + assert.equal(body.starting, true, "the response says the start is underway"); + await until(() => m.calls.create.length === 1); + const [name, spec] = m.calls.create[0]; + assert.equal(name, "pithagoras-office-ds"); + assert.equal(spec.Image, DEFAULT_TAG, "the tag comes from settings (default), not a constant"); + assert.ok((spec.Env as string[]).includes(`JWT_SECRET=${jwtSecret(b.root)}`), "the container is env'd with the boot secret"); + assert.deepEqual(JSON.parse(fs.readFileSync(path.join(b.root, "ds.json"), "utf8")), { containerId: "cid-1", origin: "http://127.0.0.1", imageTag: DEFAULT_TAG }); + await until(() => m.calls.start.length === 1); + + // a second enable is an adopt: no second create, no restart + res = await b.router.dispatch("POST", "/settings", { enabled: true }); + assert.equal((res.body as Record<string, unknown>).starting, undefined, "already enabled -> no start fired"); + await new Promise((r) => setTimeout(r, 100)); // give a stray fire-and-forget a chance to (wrongly) run + assert.equal(m.calls.create.length, 1); + assert.equal(m.calls.start.length, 1); + + // disable: awaited stop + the intent marker on disk + res = await b.router.dispatch("POST", "/settings", { enabled: false }); + assert.equal(res.statusCode, 200); + assert.deepEqual(m.calls.stop, ["pithagoras-office-ds"]); + assert.ok(fs.existsSync(path.join(b.root, "user-stopped")), "user-stopped marker file written"); + assert.equal((res.body as { settings: Record<string, unknown> }).settings.enabled, false); + assert.equal((res.body as { container: Record<string, unknown> }).container.state, "stopped"); + + // and :latest is refused with a real reason + res = await b.router.dispatch("POST", "/settings", { imageTag: "onlyoffice/documentserver:latest" }); + assert.equal(res.statusCode, 400); + assert.equal((res.body as Record<string, unknown>).code, "invalid_image_tag"); + + // there is no co-editing switch in this payload any more (the agent is a co-editor under the revision gate, + // full stop): a body carrying only the LEGACY key hits the existing no-known-setting path and changes nothing + res = await b.router.dispatch("POST", "/settings", { coEdit: true }); + assert.equal(res.statusCode, 400); + assert.match(String((res.body as Record<string, unknown>).error), /no known setting/); + const after = await b.router.dispatch("GET", "/state"); + assert.equal(after.statusCode, 200, "a legacy key in the body must not wedge the state poller"); + assert.ok(!JSON.stringify(after.body).includes("coEdit"), "the removed setting never lands in the /state payload"); + + // the audit sink got the settings change + assert.ok(b.audits.some((a) => String(a.subject).includes("settings")), "the settings change was audited"); + + live.resetLive(); + await b.cleanup(); +}); + +test("/container/status + /container/start + /container/stop drive the same lifecycle", async () => { + const m = makeDocker(); + const b = await boot(undefined, m.docker); + + let res = await b.router.dispatch("GET", "/container/status"); + assert.equal(res.statusCode, 200); + assert.equal((res.body as Record<string, unknown>).state, "stopped"); + + res = await b.router.dispatch("POST", "/container/start"); + assert.equal(res.statusCode, 200); + assert.equal((res.body as Record<string, unknown>).started, true); + assert.equal((res.body as { container: Record<string, unknown> }).container.state, "running"); + + res = await b.router.dispatch("POST", "/container/start"); // idempotent adopt + assert.equal((res.body as Record<string, unknown>).started, false); + assert.equal(m.calls.start.length, 1, "never restarted"); + + res = await b.router.dispatch("POST", "/container/stop"); + assert.equal(res.statusCode, 200); + assert.deepEqual(m.calls.stop, ["pithagoras-office-ds"]); + assert.ok(fs.existsSync(path.join(b.root, "user-stopped"))); + + live.resetLive(); + await b.cleanup(); +}); + +/* ---------------- settleKept ---------------- */ + +test("settleKept: stale index entries are pruned, orphan dirs stay, fresh trash is kept and old trash is swept", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-settle-")); + // a real doc (bytes present) + a ghost entry whose dir vanished out-of-band + an orphan dir with NO index entry + const real: store.DocMeta = { docId: "real-1", title: "Real", ext: "docx", createdAt: 1, updatedAt: 1, sizeBytes: 3, mtimeMs: 1, owner: null, grants: [] }; + const ghost: store.DocMeta = { ...real, docId: "ghost-1", title: "Gone" }; + fs.mkdirSync(path.join(root, "docs", "real-1"), { recursive: true }); + fs.writeFileSync(path.join(root, "docs", "real-1", "Real.docx"), "abc"); + fs.mkdirSync(path.join(root, "docs", "orphan-9"), { recursive: true }); // index is the source of truth: NOT added back + fs.writeFileSync(path.join(root, "index.json"), JSON.stringify({ version: 1, docs: [real, ghost] })); + + const day = 24 * 60 * 60 * 1000; + const now = Date.now(); + const oldStamp = `${now - 8 * day}-old-doc`; // > 7 days -> swept + const freshStamp = `${now - 2 * day}-fresh-doc`; // kept + fs.mkdirSync(path.join(root, ".trash", oldStamp), { recursive: true }); + fs.mkdirSync(path.join(root, ".trash", freshStamp), { recursive: true }); + fs.mkdirSync(path.join(root, ".trash", "not-a-timestamp"), { recursive: true }); // not a deleteDoc entry -> untouched + + const out = settleKept(root); + assert.equal(out.sweptTrash, 1); + assert.deepEqual(out.prunedDocs, ["ghost-1"]); + + const idx = JSON.parse(fs.readFileSync(path.join(root, "index.json"), "utf8")) as { docs: store.DocMeta[] }; + assert.deepEqual(idx.docs.map((d) => d.docId), ["real-1"], "the ghost entry is gone and the orphan dir was NOT indexed"); + assert.ok(!fs.existsSync(path.join(root, ".trash", oldStamp)), "the 8-day-old trash entry was swept"); + assert.ok(fs.existsSync(path.join(root, ".trash", freshStamp)), "the 2-day-old trash entry stays (7-day TTL)"); + assert.ok(fs.existsSync(path.join(root, ".trash", "not-a-timestamp")), "non-deleteDoc entries are untouched"); +}); + +/* ---------------- GET /ds/* (browser -> DS proxy) ---------------- */ + +const API_JS_BYTES = Buffer.from("// stand-in for the real api.js: getBasePath() walks document scripts\nwindow.DocsAPI={};"); + +/** A fake Document Server on an ephemeral 127.0.0.1 port: serves /web-apps/... with a body + content-type, + * answers a dedicated 404, and records every raw request url it received (the verbatim-pass-through proof). */ +function startFakeDs(): Promise<{ server: http.Server; port: number; urls: string[] }> { + const urls: string[] = []; + const server = http.createServer((req, res) => { + urls.push(req.url); + if (req.method === "GET" && req.url?.startsWith("/web-apps/")) { + res.writeHead(200, { "Content-Type": "application/javascript; charset=utf-8", "Cache-Control": "no-cache" }); + res.end(API_JS_BYTES); + return; + } + res.writeHead(404, { "Content-Type": "text/plain" }); + res.end("nope"); + }); + return new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", () => resolve({ server, port: (server.address() as AddressInfo).port, urls })); + }); +} + +test("/ds proxy: 200 passes the body, content-type and the RAW query string verbatim to the recorded DS origin", async () => { + const b = await boot(); + const ds = await startFakeDs(); + try { + writeDsJson(b.root, `http://127.0.0.1:${ds.port}`); + // the iframe-style query: percent-encoded chars must arrive upstream byte-for-byte (no re-encoding) + const q = "/ds/web-apps/apps/api/documents/api.js?dockey=a%2Bb~c&lang=en-US&x=1%20two"; + const res = await b.router.dispatch("GET", q); + assert.equal(res.statusCode, 200, "the upstream status passes through"); + assert.equal(Buffer.from(res.data as Buffer).compare(API_JS_BYTES), 0, "the body bytes are byte-for-byte what the DS sent"); + assert.match(String(res.headers["content-type"]), /application\/javascript/, "content-type is copied"); + assert.equal(res.headers["cache-control"], "no-cache", "cache-control is copied"); + const upstream = ds.urls.at(-1); + assert.ok(upstream !== undefined, "the fake DS was hit"); + assert.equal(upstream, q.replace(/^\/ds/, ""), "path + raw query string reached the DS untouched (no re-encoding)"); + } finally { + ds.server.closeAllConnections(); + await new Promise<void>((r) => ds.server.close(() => r())); + await b.cleanup(); + } +}); + +test("/ds proxy: an upstream 404 passes through as a 404 with the upstream body", async () => { + const b = await boot(); + const ds = await startFakeDs(); + try { + writeDsJson(b.root, `http://127.0.0.1:${ds.port}`); + const res = await b.router.dispatch("GET", "/ds/missing/page.html?dockey=z"); + assert.equal(res.statusCode, 404, "the upstream status passes through"); + assert.match(String(res.data), /nope/, "the upstream body passes through"); + } finally { + ds.server.closeAllConnections(); + await new Promise<void>((r) => ds.server.close(() => r())); + await b.cleanup(); + } +}); + +test("/ds proxy: an identity request answered with GZIP bytes + LYING headers still delivers the FULL original payload", async () => { + // The exact PR #307 bug, replayed against a hostile upstream: it honours NOTHING -- it answers our + // accept-encoding: identity ask with a gzip body while declaring content-encoding: gzip and a + // content-length sized to the COMPRESSED bytes. undici auto-decodes the gzip; the handler must not copy + // either lying header, so the client reads every original byte intact -- no truncation, no manual gunzip. + const ORIGINAL = Buffer.concat(Array.from({ length: 64 }, () => API_JS_BYTES)); + assert.ok(ORIGINAL.length > 1024, "sanity: the payload is big enough for gzip to shrink it"); + const gzipped = zlib.gzipSync(ORIGINAL); + assert.ok(gzipped.length < ORIGINAL.length, "sanity: the gzip body is really smaller than the original"); + const b = await boot(); + const seenEncodings: string[] = []; + let server: http.Server | undefined; + try { + server = http.createServer((req, res) => { + seenEncodings.push(req.headers["accept-encoding"] ?? ""); + res.writeHead(200, { "Content-Type": "application/javascript; charset=utf-8", "Content-Encoding": "gzip", "Content-Length": gzipped.length }); + res.end(gzipped); + }); + await new Promise<void>((resolve, reject) => { server!.once("error", reject); server!.listen(0, "127.0.0.1", () => resolve()); }); + const port = (server.address() as AddressInfo).port; + writeDsJson(b.root, `http://127.0.0.1:${port}`); + const res = await b.router.dispatch("GET", "/ds/web-apps/apps/api/documents/api.js"); + assert.equal(res.statusCode, 200, "the upstream status passes through"); + assert.equal(seenEncodings.at(-1), "identity", "the proxy asks the DS for an UNENCODED body"); + assert.match(String(res.headers["content-type"]), /application\/javascript/, "the content-type passes through"); + assert.equal(res.headers["content-encoding"], undefined, "the (lying) upstream content-encoding is NOT copied"); + assert.ok(!Object.keys(res.headers).includes("content-length"), "no upstream-declared content-length is forwarded -- Node sizes the real buffer"); + assert.equal(Buffer.from(res.data as Buffer).compare(ORIGINAL), 0, "the FULL original payload arrives intact (undici decoded it; no truncation at the lying length)"); + } finally { + if (server) { server.closeAllConnections(); await new Promise<void>((r) => server.close(() => r())); } + await b.cleanup(); + } +}); + +test("/ds proxy: a dead origin -> 503 {code:'ds_unavailable'} and a log line", async () => { + const b = await boot(); + try { + // an ephemeral port nothing listens on: bind one, close it, reuse the number for the dead target + const dead = http.createServer(); + await new Promise<void>((resolve, reject) => { dead.once("error", reject); dead.listen(0, "127.0.0.1", () => resolve()); }); + const port = (dead.address() as AddressInfo).port; + await new Promise<void>((r) => dead.close(() => r())); + writeDsJson(b.root, `http://127.0.0.1:${port}`); + + const res = await b.router.dispatch("GET", "/ds/web-apps/apps/api/documents/api.js"); + assert.equal(res.statusCode, 503); + assert.deepEqual(res.body, { error: `the document server at http://127.0.0.1:${port} did not answer`, code: "ds_unavailable" }); + assert.ok(b.logs.some((l) => l.includes("GET /ds/") && l.includes(`http://127.0.0.1:${port}`)), "the failure left a log line"); + } finally { + await b.cleanup(); + } +}); + +test("/ds proxy: an unconfigured data dir -> 503 {code:'ds_not_configured'}; a container: origin -> the same 422 /editor-config refusal", async () => { + const b = await boot(); // fresh root, no ds.json + try { + const none = await b.router.dispatch("GET", "/ds/web-apps/apps/api/documents/api.js"); + assert.equal(none.statusCode, 503); + assert.equal((none.body as Record<string, unknown>).code, "ds_not_configured"); + + writeDsJson(b.root, "container:cid-ab12"); + const cc = await b.router.dispatch("GET", "/ds/web-apps/apps/api/documents/api.js"); + assert.equal(cc.statusCode, 422); + const body = cc.body as Record<string, unknown>; + assert.equal(body.code, "unresolvable_ds_origin"); + assert.equal(body.origin, "container:cid-ab12", "same refusal family as /editor-config"); + } finally { + await b.cleanup(); + } +}); + +/** A fake engine.io polling endpoint: answers the handshake GET and records every POST body it received. + * The real docserver's /<app>/10.0.0-<buildhash>/doc/<key>/c/ speaks exactly this shape (replayed live). */ +function startFakeEngineIo(holdMs = 0): Promise<{ server: http.Server; port: number; posts: string[] }> { + const posts: string[] = []; + const HANDSHAKE = '0{"sid":"Zkx0F6Y_6ZptlT-cAAKS","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":20000,"maxPayload":100000000}'; + const server = http.createServer((req, res) => { + if (req.method === "GET" && req.url?.startsWith("/c/?")) { + if (holdMs > 0) setTimeout(() => { res.writeHead(200, { "Content-Type": "text/plain" }); res.end(HANDSHAKE); }, holdMs); + else { res.writeHead(200, { "Content-Type": "text/plain" }); res.end(HANDSHAKE); } + return; + } + if (req.method === "POST" && req.url?.startsWith("/c/?")) { + let b = ""; + req.on("data", (d: string) => (b += d)); + req.on("end", () => { posts.push(b); res.writeHead(200, { "Content-Type": "text/plain" }); res.end(""); }); + return; + } + res.writeHead(404); res.end("nope"); + }); + return new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", () => resolve({ server, port: (server.address() as AddressInfo).port, posts })); + }); +} + +test("/ds proxy: a POST forwards method + raw body verbatim to the DS (the socket.io client channel)", async () => { + // The editor app's doc channel: engine.io GETs handshake, then its packets (the "40" connect packet and on + // every state change) arrive as POSTs to the same /c/ path. A GET-only proxy answered those with the core's + // 404 and the channel never opened -- the white-spinner root cause (reproduced live against DS 10.0). + const ds = await startFakeEngineIo(); + const b = await boot(); + try { + writeDsJson(b.root, `http://127.0.0.1:${ds.port}`); + const res = await b.router.dispatch("POST", "/ds/c/?EIO=4&transport=polling&sid=Zkx0F6Y_6ZptlT-cAAKS", "40", { "content-type": "text/plain;charset=UTF-8" }); + assert.equal(res.statusCode, 200, "the upstream answer passes back through the proxy"); + assert.deepEqual(ds.posts, ["40"], "the raw engine.io packet arrived at the DS byte-for-byte with its query string"); + } finally { + ds.server.closeAllConnections(); + await new Promise<void>((r) => ds.server.close(() => r())); + await b.cleanup(); + } +}); + +test("/ds proxy: an engine.io handshake loses its websocket upgrade offer (this proxy is polling-only)", async () => { + // fetch cannot relay WebSocket frames, so advertising upgrades:["websocket"] would only end in failed-upgrade + // churn; the handler strips it from outbound handshakes and pins the client to long-polling. + const ds = await startFakeEngineIo(); + const b = await boot(); + try { + writeDsJson(b.root, `http://127.0.0.1:${ds.port}`); + const res = await b.router.dispatch("GET", "/ds/c/?EIO=4&transport=polling"); + assert.equal(res.statusCode, 200); + const raw = String(res.data as Buffer | string); + assert.ok(raw.startsWith("0{"), "the EIO handshake prefix is preserved"); + const hs = JSON.parse(raw.slice(1)) as Record<string, unknown>; + assert.equal(hs.sid, "Zkx0F6Y_6ZptlT-cAAKS", "everything else in the handshake is untouched"); + assert.ok(!Array.isArray(hs.upgrades) || !hs.upgrades.includes("websocket"), "no websocket upgrade is offered to the client"); + } finally { + ds.server.closeAllConnections(); + await new Promise<void>((r) => ds.server.close(() => r())); + await b.cleanup(); + } +}); + +test("/ds proxy: an idle long-poll HELD past the old 15s cutoff still comes back (engine.io keeps polls open until it has data)", async () => { + // The pre-fix handler aborted every upstream fetch after exactly 15s -- and undici's abort also closes the + // upstream socket, which docserver logs as a 499 at precisely that moment. engine.io pings at least every + // 25s, so a healthy poll can sit empty for longer than 15s; the window must cover a full ping cycle. + const ds = await startFakeEngineIo(16_500); // held open just past the old 15_000ms cutoff + const b = await boot(); + try { + writeDsJson(b.root, `http://127.0.0.1:${ds.port}`); + const t0 = Date.now(); + const res = await b.router.dispatch("GET", "/ds/c/?EIO=4&transport=polling"); + assert.equal(res.statusCode, 200, "the held poll answers instead of a ds_unavailable abort"); + assert.match(String(res.data as Buffer | string), /Zkx0F6Y_6ZptlT-cAAKS/, "the handshake body arrives after the hold"); + assert.ok(Date.now() - t0 >= 15_000, "it really waited out the old cutoff window before answering"); + } finally { + ds.server.closeAllConnections(); + await new Promise<void>((r) => ds.server.close(() => r())); + await b.cleanup(); + } +}); + + diff --git a/extensions/office/test/settings.test.mts b/extensions/office/test/settings.test.mts new file mode 100644 index 000000000..a57207cd7 --- /dev/null +++ b/extensions/office/test/settings.test.mts @@ -0,0 +1,71 @@ +/** + * Unit tests for src/settings.ts: defaults when missing, field-by-field merge of a partial file, corrupt JSON + * -> defaults, and save round-trips that keep untouched fields. All roots are mkdtemp tmp dirs -- portal() is + * never reached (every call passes an explicit root). + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { loadSettings, saveSettings, DEFAULT_SETTINGS } from "../src/settings"; + +function tmpRoot(): string { + return fs.mkdtempSync(path.join(os.tmpdir(), "office-settings-")); +} + +test("loadSettings: missing file -> defaults", () => { + const root = tmpRoot(); + assert.deepEqual(loadSettings(root), DEFAULT_SETTINGS); +}); + +test("loadSettings: partial file merges field by field, unknown fields ignored", () => { + const root = tmpRoot(); + fs.writeFileSync(path.join(root, "settings.json"), JSON.stringify({ enabled: true, imageTag: "some/other:9.9", nonsense: 42 })); + assert.deepEqual(loadSettings(root), { enabled: true, imageTag: "some/other:9.9" }); +}); + +test("loadSettings: a LEGACY file carrying the removed \"coEdit\" key loads without crashing and simply ignores it", () => { + const root = tmpRoot(); + fs.writeFileSync(path.join(root, "settings.json"), JSON.stringify({ enabled: true, imageTag: "some/other:9.9", coEdit: true })); + assert.deepEqual(loadSettings(root), { enabled: true, imageTag: "some/other:9.9" }, "legacy key tolerated and not surfaced"); + // and saving over the legacy file rewrites it clean (the removed key is dropped, never echoed back) + const saved = saveSettings({ enabled: false }, root); + assert.deepEqual(saved, { enabled: false, imageTag: "some/other:9.9" }); + assert.ok(!JSON.stringify(fs.readFileSync(path.join(root, "settings.json"), "utf8")).includes("coEdit")); +}); + +test("loadSettings: corrupt JSON -> defaults; non-object JSON -> defaults", () => { + const root = tmpRoot(); + fs.writeFileSync(path.join(root, "settings.json"), "{ not json !"); + assert.deepEqual(loadSettings(root), DEFAULT_SETTINGS); + fs.writeFileSync(path.join(root, "settings.json"), JSON.stringify([1, 2, 3])); + assert.deepEqual(loadSettings(root), DEFAULT_SETTINGS); +}); + +test("loadSettings: wrong-type values fall back to the default for that field", () => { + const root = tmpRoot(); + fs.writeFileSync(path.join(root, "settings.json"), JSON.stringify({ enabled: "yes" })); + const s = loadSettings(root); + assert.equal(s.enabled, DEFAULT_SETTINGS.enabled); + assert.equal(s.imageTag, DEFAULT_SETTINGS.imageTag); +}); + +test("saveSettings: round-trip keeps other fields and ignores unknown patch keys", () => { + const root = tmpRoot(); + const first = saveSettings({ enabled: true }, root); + assert.deepEqual(first, { ...DEFAULT_SETTINGS, enabled: true }); + // a later patch must not clobber the earlier one; keys outside OfficeSettings (incl. the legacy + // "coEdit") are dropped at runtime and never persisted back + const second = saveSettings({ imageTag: "pinned/x:2.0", bogusKey: "x" } as Partial<{ enabled: boolean; imageTag: string; coEdit?: boolean }>, root); + assert.deepEqual(second, { enabled: true, imageTag: "pinned/x:2.0" }); + assert.deepEqual(loadSettings(root), second); +}); + +test("saveSettings: overwrites a corrupt file with clean defaults", () => { + const root = tmpRoot(); + fs.writeFileSync(path.join(root, "settings.json"), "{{{"); + const s = saveSettings({ imageTag: "x/y:1.2" }, root); + assert.equal(s.imageTag, "x/y:1.2"); + assert.equal(s.enabled, DEFAULT_SETTINGS.enabled); +}); diff --git a/extensions/office/test/store.test.mts b/extensions/office/test/store.test.mts new file mode 100644 index 000000000..dff0e068f --- /dev/null +++ b/extensions/office/test/store.test.mts @@ -0,0 +1,524 @@ +/** + * Unit tests for src/store.ts: the document library (index + docs/<docId>/), CRUD, ownership/grant scoping, + * import/export, the persistBytes REVISION GATE and trash sweeping. Templates are asserted to be REAL office + * files by parsing them with src/ooxml.ts (parseDocx/parseXlsx). Every root is an mkdtemp dir in os.tmpdir() -- + * the real data dir and portal() are never touched. + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import JSZip from "jszip"; +import { + listDocs, getDoc, docsForHome, createDoc, renameDoc, deleteDoc, sweepTrash, + importFile, exportDoc, persistBytes, templateDocx, templateXlsx, slugify, + scanWorkspace, pruneMissingDocs, +} from "../src/store"; +import { parseDocx, parseXlsx } from "../src/ooxml"; + +function tmpRoot(): string { + return fs.mkdtempSync(path.join(os.tmpdir(), "office-store-")); +} + +const docFile = (root: string, meta: { docId: string; title: string; ext: string }) => path.join(root, "docs", meta.docId, `${meta.title}.${meta.ext}`); + +/* ---------------- templates ---------------- */ + +test("templateDocx: minimal docx parses with ooxml.parseDocx and carries real content", async () => { + const buf = await templateDocx(); + assert.ok(buf.length > 500); + const parsed = parseDocx(buf); // throws if the zip or word/document.xml is missing/broken + assert.deepEqual(parsed.paragraphs, ["New document"]); + assert.equal(parsed.comments.length, 0); + // every part the contract names must be present in the archive + const zip = await JSZip.loadAsync(buf); + for (const name of ["[Content_Types].xml", "_rels/.rels", "word/document.xml", "word/styles.xml"]) assert.ok(zip.file(name), `missing ${name}`); +}); + +test("templateXlsx: minimal xlsx parses with ooxml.parseXlsx as one empty Sheet1", async () => { + const buf = await templateXlsx(); + const parsed = parseXlsx(buf); // throws if workbook/rels/sheet parts are missing/broken + assert.deepEqual(parsed.sheetNames, ["Sheet1"]); + assert.deepEqual(parsed.sheets["Sheet1"], []); + const zip = await JSZip.loadAsync(buf); + for (const name of ["[Content_Types].xml", "_rels/.rels", "xl/workbook.xml", "xl/_rels/workbook.xml.rels", "xl/worksheets/sheet1.xml"]) assert.ok(zip.file(name), `missing ${name}`); +}); + +test("slugify: lowercase, non-alnum runs -> single dash, trimmed", () => { + assert.equal(slugify("Quarterly Report Q3"), "quarterly-report-q3"); + assert.equal(slugify(" --Weird__Name!! "), "weird-name"); +}); + +/* ---------------- create / list / get ---------------- */ + +test("createDoc: word and cell write parseable templates into docs/<docId>/", async () => { + const root = tmpRoot(); + const w = await createDoc({ title: "Quarterly Report", type: "word" }, root); + assert.equal(w.ext, "docx"); + assert.match(w.docId, /^quarterly-report-[a-z0-9]{6}$/); + assert.equal(parseDocx(fs.readFileSync(docFile(root, w))).text, "New document"); + assert.ok(w.sizeBytes > 0 && w.mtimeMs > 0); + + const c = await createDoc({ title: "Budget", type: "cell" }, root); + assert.equal(c.ext, "xlsx"); + assert.match(c.docId, /^budget-[a-z0-9]{6}$/); + assert.deepEqual(parseXlsx(fs.readFileSync(docFile(root, c))).sheetNames, ["Sheet1"]); + + // fresh index.json on disk, version 1 + const index = JSON.parse(fs.readFileSync(path.join(root, "index.json"), "utf8")); + assert.equal(index.version, 1); + assert.equal(index.docs.length, 2); +}); + +test("createDoc: duplicate (same title + type) throws code 'exists'; same title different type is fine", async () => { + const root = tmpRoot(); + await createDoc({ title: "Report", type: "word" }, root); + await assert.rejects(async () => createDoc({ title: "Report", type: "word" }, root), (err: Error) => err.code === "exists"); + // a spreadsheet with the same title is a different document, not a duplicate + const cell = await createDoc({ title: "Report", type: "cell" }, root); + assert.equal(cell.ext, "xlsx"); +}); + +test("listDocs: titles ascending; getDoc miss -> null", async () => { + const root = tmpRoot(); + const banana = await createDoc({ title: "Banana Plan", type: "word" }, root); + const apple = await createDoc({ title: "apple notes", type: "word" }, root); + assert.deepEqual(listDocs(root).map((d) => d.docId), [apple.docId, banana.docId]); + assert.equal(getDoc("nope-123456", root), null); + const found = getDoc(apple.docId, root); + assert.ok(found && found.title === "apple notes"); +}); + +/* ---------------- ownership / grants ---------------- */ + +test("docsForHome: owner path, grant path, dedupe of owned-and-granted doc; others' docs excluded", async () => { + const root = tmpRoot(); + const homeA = "chat-A"; + const homeB = "chat-B"; + const owned = await createDoc({ title: "A owned doc", owner: homeA }, root); + const granted = await createDoc({ title: "B doc granted to A", owner: homeB }, root); + const stranger = await createDoc({ title: "C doc nobody shares", owner: "chat-C" }, root); + + // grant A access to the B doc (what ping.ts does on a mention) + const index = JSON.parse(fs.readFileSync(path.join(root, "index.json"), "utf8")); + const entry = index.docs.find((d: { docId: string }) => d.docId === granted.docId); + entry.grants.push({ chatId: homeA, commentId: "c1", at: Date.now() }); + // the A-owned doc is ALSO granted to A: dedupe must keep it exactly once + const ownedEntry = index.docs.find((d: { docId: string }) => d.docId === owned.docId); + ownedEntry.grants.push({ chatId: homeA, commentId: "c2", at: Date.now() }); + fs.writeFileSync(path.join(root, "index.json"), JSON.stringify(index)); + + const visible = docsForHome(homeA, root).map((d) => d.docId); + assert.deepEqual(visible.sort(), [granted.docId, owned.docId].sort()); + assert.equal(visible.filter((id) => id === owned.docId).length, 1); // deduped + assert.ok(!visible.includes(stranger.docId)); + + const bView = docsForHome(homeB, root).map((d) => d.docId); + assert.deepEqual(bView, [granted.docId]); +}); + +test("docsForHome: UNOWNED docs (workspace-scanned and human-made) are visible to EVERY home", async () => { + const root = tmpRoot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-unowned-")); + fs.writeFileSync(path.join(ws, "shared plan.docx"), Buffer.from("a shared docx of sorts")); + assert.deepEqual(scanWorkspace([ws], root), { added: 1, updated: 0 }); + const scanned = getDoc(listDocs(root)[0].docId, root)!; + assert.equal(scanned.owner, null); + + // a human-made doc (no owner passed) is shared too, while another home's OWNED doc stays private to it + const human = await createDoc({ title: "Human-made" }, root); + assert.equal(human.owner, null); + const onlyB = await createDoc({ title: "B only", owner: "chat-B" }, root); + + for (const home of ["chat-A", "chat-C"]) { + const visible = docsForHome(home, root).map((d) => d.docId); + assert.ok(visible.includes(scanned.docId), `${home} sees the unowned scanned doc`); + assert.ok(visible.includes(human.docId), `${home} sees the unowned human doc`); + assert.ok(!visible.includes(onlyB.docId), `${home} does NOT see chat-B's owned doc`); + } + const bView = docsForHome("chat-B", root).map((d) => d.docId); + assert.deepEqual(bView.sort(), [scanned.docId, human.docId, onlyB.docId].sort()); + fs.rmSync(ws, { recursive: true, force: true }); +}); + +/* ---------------- rename / delete / trash ---------------- */ + +test("renameDoc: renames the file on disk, keeps bytes and mtimeMs, bumps updatedAt", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Old Title" }, root); + const oldBytes = fs.readFileSync(docFile(root, doc)); + const before = getDoc(doc.docId, root); + assert.ok(before); + + const renamed = renameDoc(doc.docId, "New Title", root); + assert.equal(renamed.docId, doc.docId); // stable across renames -- tickets/history key off it + assert.equal(renamed.title, "New Title"); + assert.ok(!fs.existsSync(docFile(root, { ...doc, title: "Old Title" }))); + const newBytes = fs.readFileSync(docFile(root, renamed)); + assert.deepEqual(newBytes, oldBytes); // bytes untouched + assert.equal(renamed.mtimeMs, before!.mtimeMs); // content mtime (the revision) is preserved by the rename + assert.ok(renamed.updatedAt >= before!.updatedAt); + // renaming onto another doc's title -> exists + await createDoc({ title: "Taken", type: "word" }, root); + assert.throws(() => renameDoc(doc.docId, "Taken", root), (err: Error) => err.code === "exists"); +}); + +test("renameDoc on unknown id -> not_found", () => { + const root = tmpRoot(); + assert.throws(() => renameDoc("ghost-123456", "X", root), (err: Error) => err.code === "not_found"); +}); + +test("deleteDoc: moves docs/<docId> into .trash, prunes the index; unknown id -> not_found", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Doomed", type: "word" }, root); + deleteDoc(doc.docId, root); + assert.ok(!fs.existsSync(path.join(root, "docs", doc.docId))); + const trash = fs.readdirSync(path.join(root, ".trash")); + assert.equal(trash.length, 1); + assert.match(trash[0], new RegExp(`^\\d+-${doc.docId}$`)); + assert.ok(fs.existsSync(path.join(root, ".trash", trash[0], "Doomed.docx"))); // content kept for the TTL + assert.equal(getDoc(doc.docId, root), null); + assert.throws(() => deleteDoc(doc.docId, root), (err: Error) => err.code === "not_found"); +}); + +test("sweepTrash: removes only entries older than 7 days", () => { + const root = tmpRoot(); + const trash = path.join(root, ".trash"); + const oldStamp = Date.now() - 8 * 24 * 60 * 60 * 1000; // 8 days + fs.mkdirSync(path.join(trash, `${oldStamp}-ancient-doc`), { recursive: true }); + fs.writeFileSync(path.join(trash, `${oldStamp}-ancient-doc`, "x.docx"), Buffer.from("z")); + const freshStamp = Date.now() - 3600 * 1000; // 1h ago + fs.mkdirSync(path.join(trash, `${freshStamp}-young-doc`), { recursive: true }); + fs.writeFileSync(path.join(trash, `${freshStamp}-young-doc`, "x.docx"), Buffer.from("z")); + fs.mkdirSync(path.join(trash, "not-a-stamped-entry"), { recursive: true }); // no numeric prefix -- leave it alone + + const removed = sweepTrash(root); + assert.equal(removed, 1); + assert.ok(!fs.existsSync(path.join(trash, `${oldStamp}-ancient-doc`))); + assert.ok(fs.existsSync(path.join(trash, `${freshStamp}-young-doc`))); + assert.ok(fs.existsSync(path.join(trash, "not-a-stamped-entry"))); + + // empty root: no .trash at all -> 0, no throw + assert.equal(sweepTrash(tmpRoot()), 0); +}); + +/* ---------------- import / export ---------------- */ + +test("importFile: copies a real docx in (title from basename by default), records size and mtime", async () => { + const root = tmpRoot(); + const srcRoot = tmpRoot(); + const template = await templateDocx(); + const src = path.join(srcRoot, "Client proposal FINAL.docx"); + fs.writeFileSync(src, template); + + const meta = importFile(src, {}, root); + assert.equal(meta.title, "Client proposal FINAL"); + assert.equal(meta.ext, "docx"); + assert.match(meta.docId, /^client-proposal-final-[a-z0-9]{6}$/); + const stored = fs.readFileSync(docFile(root, meta)); + assert.deepEqual(stored, template); + assert.equal(parseDocx(stored).text, "New document"); + assert.deepEqual(meta.importedBaseline, []); // parseable source without comments -> empty baseline stamped at import time + assert.equal(meta.sizeBytes, template.length); + + // explicit title + owner override the defaults + const src2 = path.join(srcRoot, "grid.xlsx"); + fs.writeFileSync(src2, await templateXlsx()); + const meta2 = importFile(src2, { title: "Grid", owner: "chat-9" }, root); + assert.equal(meta2.title, "Grid"); + assert.equal(meta2.owner, "chat-9"); +}); + +test("importFile: .txt -> unsupported_format; missing file -> not_found; uppercase extension accepted", async () => { + const root = tmpRoot(); + const srcRoot = tmpRoot(); + const txt = path.join(srcRoot, "notes.txt"); + fs.writeFileSync(txt, "hi"); + await assert.rejects(async () => importFile(txt, {}, root), (err: Error) => err.code === "unsupported_format"); + await assert.rejects(async () => importFile(path.join(srcRoot, "ghost.docx"), {}, root), (err: Error) => err.code === "not_found"); + + const upper = path.join(srcRoot, "UPPER.DOCX"); + fs.writeFileSync(upper, await templateDocx()); + const meta = importFile(upper, {}, root); + assert.equal(meta.ext, "docx"); // normalized to the stored extension +}); + +test("importFile: a file above 512 MiB is rejected with too_large before anything is copied", async () => { + const root = tmpRoot(); + const srcRoot = tmpRoot(); + // Sparse on purpose: ftruncate advertises the size without writing 600 MiB of zeros (stat-based cap). + const big = path.join(srcRoot, "huge.docx"); + const fd = fs.openSync(big, "w"); + try { + fs.ftruncateSync(fd, 600 * 1024 * 1024); + } finally { + fs.closeSync(fd); + } + await assert.rejects(async () => importFile(big, {}, root), (err: Error) => err.code === "too_large" && /512 MiB/.test(err.message)); + assert.equal(listDocs(root).length, 0, "nothing landed in the library"); +}); + +test("importFile: docx at/under 32 MiB stamps importedBaseline; above it leaves it undefined (ping fallback)", async () => { + const root = tmpRoot(); + const srcRoot = tmpRoot(); + + // under the cap: a real parseable docx with one comment -> baseline of its ids (the existing rule) + const small = path.join(srcRoot, "small.docx"); + fs.writeFileSync(small, await templateDocx()); + const metaSmall = importFile(small, {}, root); + assert.deepEqual(metaSmall.importedBaseline, [], "under-cap docx still parses and baselines (no comments in the template)"); + + // above the cap: a sparse 33 MiB .docx -- no parsing happens at all, baseline unknown, first-save fallback applies + const big = path.join(srcRoot, "big.docx"); + const fd = fs.openSync(big, "w"); + try { + fs.ftruncateSync(fd, 33 * 1024 * 1024); + } finally { + fs.closeSync(fd); + } + const metaBig = importFile(big, {}, root); + assert.equal(metaBig.importedBaseline, undefined, "above-cap docx skips the baseline; ping.ts baselines everything at first save"); + assert.ok(metaBig.sizeBytes > 32 * 1024 * 1024); +}); + +test("exportDoc: writes <title>.<ext> into a created dest dir and returns the absolute path", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Export me", type: "word" }, root); + const destDir = path.join(tmpRoot(), "out", "nested"); // must be created recursively + const written = exportDoc(doc.docId, destDir, root); + assert.equal(written, path.join(destDir, "Export me.docx")); + assert.ok(path.isAbsolute(written)); + assert.deepEqual(parseDocx(fs.readFileSync(written)), parseDocx(fs.readFileSync(docFile(root, doc)))); + assert.throws(() => exportDoc("ghost-123456", destDir, root), (err: Error) => err.code === "not_found"); +}); + +/* ---------------- persist gate ---------------- */ + +test("persistBytes: success writes the bytes, updates the index mtime/size and returns the new mtimeMs", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Live" }, root); + const oldMeta = getDoc(doc.docId, root); + assert.ok(oldMeta); + const saved = Buffer.from("saved-by-DS-" + "x".repeat(100)); + + const res = persistBytes(oldMeta, saved, root); + const fresh = fs.statSync(docFile(root, doc)); + assert.equal(res.mtimeMs, fresh.mtimeMs); + const updated = getDoc(doc.docId, root); + assert.ok(updated); + assert.equal(updated!.mtimeMs, res.mtimeMs); + assert.equal(updated!.sizeBytes, saved.length); + assert.deepEqual(fs.readFileSync(docFile(root, doc)), saved); // the bytes really landed +}); + +test("persistBytes: a stale meta (external write moved the mtime) throws revision_mismatch with currentMtimeMs", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Contended" }, root); + const staleMeta = getDoc(doc.docId, root); + assert.ok(staleMeta); + + // an external writer (another DS session / the agent) rewrites the file and pushes its mtime far forward + const file = docFile(root, doc); + fs.writeFileSync(file, Buffer.from("someone else saved this")); + const forced = new Date(Date.now() + 60_000).getTime(); // a definitely-distinct mtime + fs.utimesSync(file, forced / 1000, forced / 1000); + const currentMtimeMs = fs.statSync(file).mtimeMs; + assert.notEqual(currentMtimeMs, staleMeta.mtimeMs); + + await assert.rejects( + (async () => persistBytes(staleMeta, Buffer.from("my stale save"), root))(), + (err: Error) => { + assert.equal(err.code, "revision_mismatch"); + assert.equal((err as NodeJS.ErrnoException & { currentMtimeMs?: number }).currentMtimeMs, currentMtimeMs); + return true; + }, + ); + // the library file is untouched by the refused write + assert.deepEqual(fs.readFileSync(file), Buffer.from("someone else saved this")); +}); + +test("persistBytes: an FRESH meta after the external write persists again (the gate clears)", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Retry" }, root); + const file = docFile(root, doc); + fs.writeFileSync(file, Buffer.from("newer content")); + const freshMeta = getDoc(doc.docId, root) ?? undefined; // re-read: still the OLD index mtime -> would mismatch + assert.ok(freshMeta); + // but persistBytes with the stale index meta must fail... + await assert.rejects((async () => persistBytes(freshMeta, Buffer.from("z"), root))(), (err: Error) => err.code === "revision_mismatch"); +}); + +/* ---------------- end-to-end-ish ---------------- */ + +test("a full round-trip: create -> edit bytes through the gate -> export is parseable", async () => { + const root = tmpRoot(); + const doc = await createDoc({ title: "Round Trip" }, root); + let meta = getDoc(doc.docId, root)!; + for (let i = 0; i < 3; i++) { + const bytes = Buffer.from(`revision ${i} of the round trip document`); + await new Promise((r) => setTimeout(r, 2)); // let mtime advance so successive persists are distinct revisions + const res = persistBytes(meta, bytes, root); + meta = getDoc(doc.docId, root)!; + assert.equal(meta.mtimeMs, res.mtimeMs); + } + const out = exportDoc(doc.docId, path.join(tmpRoot(), "rt"), root); + assert.equal(fs.readFileSync(out).toString("utf8"), "revision 2 of the round trip document"); +}); + +/* ---------------- workspace scan (in-place indexing, replaces manual import) ---------------- */ + +test("scanWorkspace: discovers nested docx/xlsx case-insensitively; skips hidden dirs, node_modules, .txt", () => { + const root = tmpRoot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-scan-ws-")); + fs.writeFileSync(path.join(ws, "alpha.docx"), Buffer.from("bytes a")); + fs.mkdirSync(path.join(ws, "nested", "deeper"), { recursive: true }); + fs.writeFileSync(path.join(ws, "nested", "deeper", "BETA.XLSX"), Buffer.from("bytes b")); // uppercase ext counts + fs.mkdirSync(path.join(ws, ".cache")); + fs.writeFileSync(path.join(ws, ".cache", "hidden.docx"), Buffer.from("nope")); + fs.mkdirSync(path.join(ws, "node_modules", "pkg"), { recursive: true }); + fs.writeFileSync(path.join(ws, "node_modules", "pkg", "vendor.docx"), Buffer.from("nope")); + fs.writeFileSync(path.join(ws, "readme.txt"), "not office"); + + const out = scanWorkspace([ws], root); + assert.deepEqual(out, { added: 2, updated: 0 }); + const docs = listDocs(root); + assert.deepEqual(docs.map((d) => d.title), ["alpha", "BETA"], "listDocs keeps its localeCompare order even with a mixed-case title"); + for (const doc of docs) { + assert.equal(doc.path, path.join(ws, doc.title === "alpha" ? "alpha.docx" : path.join("nested", "deeper", "BETA.XLSX")), "points at the real file in place"); + assert.equal(doc.owner, null); // unowned human doc: docsForHome shows it to every home + assert.ok(fs.existsSync(doc.path!)); // IN PLACE: no copy was made + } + fs.rmSync(ws, { recursive: true, force: true }); +}); + +test("scanWorkspace: never indexes the library's own dir (docs/ or .trash), even when it is passed as a root", async () => { + const root = tmpRoot(); + await createDoc({ title: "Inside The Library", type: "word" }, root); // lives in docs/<id>/ -- must stay unindexed-as-scanned + fs.mkdirSync(path.join(root, ".trash")); + fs.writeFileSync(path.join(root, ".trash", "old.docx"), Buffer.from("trash stays trash")); + + const out = scanWorkspace([root], root); // the data dir IS a root: the whole walk must refuse to enter it + assert.deepEqual(out, { added: 0, updated: 0 }); + const doc = listDocs(root)[0]; + assert.equal(doc.path, undefined, "the library-created doc has no path and is not re-indexed over itself"); +}); + +test("scanWorkspace: a second scan refreshes changed mtime/size in place; untouched files do not count as updated", () => { + const root = tmpRoot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-scan-refresh-")); + fs.writeFileSync(path.join(ws, "one.docx"), Buffer.from("12345")); + fs.writeFileSync(path.join(ws, "two.xlsx"), Buffer.from("abcdef")); + assert.deepEqual(scanWorkspace([ws], root), { added: 2, updated: 0 }); + + assert.deepEqual(scanWorkspace([ws], root), { added: 0, updated: 0 }, "immediately rescanning changes nothing"); + + fs.writeFileSync(path.join(ws, "one.docx"), Buffer.from("123456789")); // grew AND got a new mtime + const out = scanWorkspace([ws], root); + assert.deepEqual(out, { added: 0, updated: 1 }, "only the file that actually changed counts as refreshed"); + const one = getDoc(listDocs(root).find((d) => d.title === "one")!.docId, root)!; + assert.equal(one.sizeBytes, 9); + assert.ok(Math.abs(one.mtimeMs - fs.statSync(path.join(ws, "one.docx")).mtimeMs) < 50); + + // and the refreshed doc still persists through the mtime gate: its revision IS the file's own mtime + const saved = persistBytes(one, Buffer.from("123456789X"), root); + assert.equal(saved.mtimeMs, fs.statSync(path.join(ws, "one.docx")).mtimeMs); + fs.rmSync(ws, { recursive: true, force: true }); +}); + +test("scanWorkspace stamps DocMeta.workspace: first path segment under the DEEPEST scan root containing the file; a direct child takes the root's own dir name", () => { + const root = tmpRoot(); + const base = fs.mkdtempSync(path.join(os.tmpdir(), "office-wslabel-")); + const repos = path.join(base, "repos"); // an inner root (e.g. a repo dir) nested under another one + const home = path.join(base, "homes", "acme"); // a home root; the deepest root for its files + fs.mkdirSync(path.join(repos, "pithagoras", "extensions"), { recursive: true }); + fs.mkdirSync(path.join(home, "plans"), { recursive: true }); + fs.writeFileSync(path.join(repos, "pithagoras", "extensions", "sample.docx"), Buffer.from("a docx of sorts")); + fs.writeFileSync(path.join(repos, "top.xlsx"), Buffer.from("an xlsx straight in the inner root")); + fs.writeFileSync(path.join(home, "readme.docx"), Buffer.from("directly in the home root")); + fs.writeFileSync(path.join(home, "plans", "q3.xlsx"), Buffer.from("nested in the home")); + + // base is ALSO a root: the deepest matching root must label, not the broadest (via base, sample would read "repos") + assert.deepEqual(scanWorkspace([base, repos, home], root), { added: 4, updated: 0 }); + const byTitle = new Map(listDocs(root).map((d) => [d.title, d.workspace])); + assert.equal(byTitle.get("sample"), "pithagoras", "first path segment under the deepest containing root (repos, not base)"); + assert.equal(byTitle.get("top"), "repos", "a file straight in a root takes that root's OWN dir name, not its own file name"); + assert.equal(byTitle.get("readme"), "acme", "a home-root file takes the home's dir name"); + assert.equal(byTitle.get("q3"), "plans", "a nested file takes its first segment under the root"); + fs.rmSync(base, { recursive: true, force: true }); +}); + +test("scanWorkspace rescan BACKFILLS DocMeta.workspace on entries indexed before the stamp (counted as refreshed exactly once)", () => { + const root = tmpRoot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-wsbackfill-")); + fs.mkdirSync(path.join(ws, "pithagoras", "fixtures"), { recursive: true }); + fs.writeFileSync(path.join(ws, "pithagoras", "fixtures", "sample.docx"), Buffer.from("a docx of sorts")); + assert.deepEqual(scanWorkspace([ws], root), { added: 1, updated: 0 }); + + // simulate an older index.json (pre-stamp): the entry's workspace field is simply absent + const indexPathFile = path.join(root, "index.json"); + const idx = JSON.parse(fs.readFileSync(indexPathFile, "utf8")); + delete idx.docs[0].workspace; + fs.writeFileSync(indexPathFile, JSON.stringify(idx)); + assert.equal(listDocs(root)[0].workspace, undefined); + + const second = scanWorkspace([ws], root); + assert.deepEqual(second, { added: 0, updated: 1 }, "the backfill counts as a refresh, once"); + assert.equal(listDocs(root)[0].workspace, "pithagoras", "the existing entry picked up its label on the rescan"); + + const third = scanWorkspace([ws], root); + assert.deepEqual(third, { added: 0, updated: 0 }, "idempotent once backfilled"); + fs.rmSync(ws, { recursive: true, force: true }); +}); + +test("scanWorkspace: skips files whose basename cannot be a title or busts the size cap; a collision on title does not sink the sweep", () => { + const root = tmpRoot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-scan-bad-")); + fs.writeFileSync(path.join(ws, "bad\tname.docx"), Buffer.from("control char in a title: skipped, not indexed")); + fs.writeFileSync(path.join(ws, "hugemem.docx"), Buffer.alloc(1)); // size is faked below with sparse write + fs.rmSync(path.join(ws, "hugemem.docx")); + const huge = fs.openSync(path.join(ws, "hugemem.docx"), "w"); + fs.ftruncateSync(huge, 513 * 1024 * 1024); // just above the 512 MiB cap: sparse file, no real I/O + fs.closeSync(huge); + fs.writeFileSync(path.join(ws, "same.docx"), Buffer.from("first")); + const ws2 = fs.mkdtempSync(path.join(os.tmpdir(), "office-scan-bad2-")); + fs.writeFileSync(path.join(ws2, "same.docx"), Buffer.from("second -- same title in another folder: skipped by the guard")); + + const out = scanWorkspace([ws, ws2], root); + assert.deepEqual(out, { added: 1, updated: 0 }, "only 'same' from the first root survived: bad title, over cap and title collision are all quiet skips"); + assert.equal(listDocs(root)[0].title, "same"); + fs.rmSync(ws, { recursive: true, force: true }); + fs.rmSync(ws2, { recursive: true, force: true }); +}); + +test("deleteDoc on a scanned doc moves its REAL file to the office trash; an already-vanished file is a clean prune, not ENOENT", () => { + const root = tmpRoot(); + const ws = fs.mkdtempSync(path.join(os.tmpdir(), "office-scan-del-")); + fs.writeFileSync(path.join(ws, "doomed.docx"), Buffer.from("soon to be trashed")); + scanWorkspace([ws], root); + const doc = listDocs(root)[0]; + + deleteDoc(doc.docId, root); // the real workspace file is what moves + assert.ok(!fs.existsSync(path.join(ws, "doomed.docx")), "the workspace file is gone from its folder"); + const trash = fs.readdirSync(path.join(root, ".trash")); + assert.equal(trash.length, 1, "exactly one entry landed in the office trash -- recoverable for 7 days"); + assert.equal(fs.readFileSync(path.join(root, ".trash", trash[0]), "utf8"), "soon to be trashed"); + assert.equal(listDocs(root).length, 0); + + // a file deleted OUT-OF-BAND (no one asked the library): deleting its index entry must not ENOENT + fs.writeFileSync(path.join(ws, "walked.docx"), Buffer.from("leaving on its own")); + scanWorkspace([ws], root); + const gone = listDocs(root)[0]; + fs.rmSync(path.join(ws, "walked.docx")); // the workspace itself loses the file + deleteDoc(gone.docId, root); // and: no throw, just the index drop -- pruneMissingDocs semantics on demand + assert.equal(listDocs(root).length, 0); + + // and pruneMissingDocs agrees with that: it drops the orphaned entry, nothing else + fs.writeFileSync(path.join(ws, "fleeing.docx"), Buffer.from("gone before the sweep")); + scanWorkspace([ws], root); + fs.rmSync(path.join(ws, "fleeing.docx")); + const fleeingId = listDocs(root)[0].docId; + assert.deepEqual(pruneMissingDocs(root), [fleeingId], "the orphaned scanned entry is dropped by the next settleKept"); + assert.equal(listDocs(root).length, 0); + fs.rmSync(ws, { recursive: true, force: true }); +}); diff --git a/extensions/office/test/tools.test.mts b/extensions/office/test/tools.test.mts new file mode 100644 index 000000000..3d09ba711 --- /dev/null +++ b/extensions/office/test/tools.test.mts @@ -0,0 +1,431 @@ +/** + * Unit tests for src/tools.ts (the agent's office_* tools, plan 5.4/5.5). Run from the repo ROOT: + * env -u NODE_ENV node --import tsx --test extensions/office/test/tools.test.mts + * Self-contained: tmp dataDir per test, real docx/xlsx via store.createDoc (real templates), comment + * XML injected with ooxml.rewriteZip, every portal bit (extend registrar, chat->folder, folders list, + * inFolder, root, record, log) injected through registerTools -- the tests NEVER reach portal(). + * The co-editor model is under test here too: a live human session neither blocks nor helps office_edit; + * only the revision gate decides (fail loud, re-read + retry). + * + * Refusals are THROWN by the tool's execute (docs convention: "a refusal is an error the agent + * reads"), so error cases assert on the rejection MESSAGE, which must start with "error:<code> ". + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { registerTools, __resetToolsForTests } from "../src/tools"; +import { createDoc, getDoc } from "../src/store"; +import { parseDocx, rewriteZip } from "../src/ooxml"; +import { setUser, resetLive } from "../src/live"; + +/* ------------------------------------------------------------------ */ +/* fakes */ +/* ------------------------------------------------------------------ */ + +type FolderLite = { path: string; name: string; kind: "agent" | "project" | "chat"; home?: boolean }; +const FOLDERS: FolderLite[] = [ + { path: "/f/atlas", name: "Atlas", kind: "agent" }, + { path: "/f/acme", name: "Acme", kind: "project" }, + { path: "/f/notes", name: "Notes", kind: "chat" }, // wrong kind: never gets tools +]; +const HOME_BY_FOLDER: Record<string, string[]> = { + "/f/atlas": ["chat-atlas-home"], + "/f/acme": ["zzz-acme-b", "aaa-acme-a"], // two sessions; the deterministic pick must be the lowest id +}; +const CHAT_TO_FOLDER: Record<string, string> = { + "chat-atlas-home": "/f/atlas", + "aaa-acme-a": "/f/acme", + "zzz-acme-b": "/f/acme", + "chat-notes": "/f/notes", +}; + +type ToolResult = { content: { type: string; text: string }[]; details?: any }; +type PiFake = { registered: Record<string, any>[]; sent: unknown[][]; uiHandlers: ((m: any) => void)[]; find: (name: string) => any } & Record<string, any>; + +function makePi(): PiFake { + const registered: Record<string, any>[] = []; + const sent: unknown[][] = []; + const uiHandlers: ((m: any) => void)[] = []; + return Object.assign(({} as any), { + registered, + sent, + uiHandlers, + find: (name: string) => registered.find((t) => t.name === name), + registerTool: (def: Record<string, any>) => { + registered.push(def); + }, + sendMessage: (msg: unknown, opts: unknown) => { + sent.push([msg, opts]); + }, + events: { + on: (_name: string, fn: (m: any) => void) => { + uiHandlers.push(fn); + return () => {}; + }, + }, + }); +} + +type Seen = { records: { subject: string; reason: string; chatId: string }[]; logs: string[] }; + +/** One isolated world: tmp root, a captured extend factory, a fake pi for `chatId`, seams wired so no default touches portal(). */ +function setup(chatId = "chat-atlas-home"): { root: string; seen: Seen; pi: PiFake; folderPath: string } { + __resetToolsForTests(); + resetLive(); + const root = fs.mkdtempSync(path.join(os.tmpdir(), "office-tools-")); + const seen: Seen = { records: [], logs: [] }; + const pi = makePi(); + let factory!: (pi: unknown, chat: { id: string }) => void; + registerTools({ + extend: (f) => { + factory = f; + }, + chatFolder: (id) => { + const p = CHAT_TO_FOLDER[id]; + if (!p) throw new Error(`no workspace for ${id}`); + return p; + }, + folders: () => FOLDERS, + inFolder: (p) => (HOME_BY_FOLDER[p] ?? []).map((id) => ({ id })), + root, + record: (e) => seen.records.push(e), + log: (l) => seen.logs.push(l), + }); + // The factory body runs PER CALL so each test step sees exactly one registration pass for its chat. + (setup as any).__run = () => factory(pi, { id: chatId }); + return { root, seen, pi, folderPath: CHAT_TO_FOLDER[chatId] }; +} + +/** Run the captured factory on the fake pi and hand back the registered tool by name. */ +function tool(s: ReturnType<typeof setup>, name: string): any { + (setup as any).__run(); + const t = s.pi.find(name); + if (!t) throw new Error(`tool ${name} not registered`); + s.pi.registered.length = 0; // registrations must not stack across calls on the same pi + return t; +} + +/** Await an execute that MUST refuse, and return the refusal message (docs convention: it is thrown, and starts with "error:<code> "). */ +async function refused(p: Promise<unknown>, what = ""): Promise<string> { + try { + const v = await p; + throw new Error(`expected a refusal (${what}) but the tool succeeded: ${JSON.stringify((v as ToolResult)?.content ?? v)}`); + } catch (e) { + if (e instanceof Error && e.message.startsWith("expected a refusal")) throw e; + const msg = e instanceof Error ? e.message : String(e); + if (!/^error:[a-z_]+ /.test(msg)) throw new Error(`bad refusal format (${what}): ${msg}`); + return msg; + } +} + +const readDocBytes = (root: string, docId: string): Buffer => { + const doc = getDoc(docId, root)!; + return fs.readFileSync(path.join(root, "docs", docId, `${doc.title}.${doc.ext}`)); +}; + +const esc = (s: string) => s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">"); +function commentsXml(cs: { id: string; author: string; text: string }[]): string { + return ( + '<w:comments xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">' + + cs.map((c) => `<w:comment id="${c.id}" author="${esc(c.author)}" date="2026-01-02T03:04:05Z"><w:p><w:r><w:t xml:space="preserve">${esc(c.text)}</w:t></w:r></w:p></w:comment>`).join("") + + "</w:comments>" + ); +} + +function addGrant(root: string, docId: string, chatId: string, commentId: string): void { + const p = path.join(root, "index.json"); + const raw = JSON.parse(fs.readFileSync(p, "utf8")); + raw.docs.find((d: any) => d.docId === docId).grants.push({ chatId, commentId, at: Date.now() }); + fs.writeFileSync(p, JSON.stringify(raw, null, 2)); +} + +/** Simulate an EXTERNAL save -- a co-editor's save landing on disk through the DS callback persist path -- by moving the file mtime AND the index entry together (so every gate sees one consistent fresh revision). + * NOTE: this guest filesystem reports utimes'd timestamps ~1 µs BELOW the requested value, so the index is set to + * what stat ACTUALLY reads back -- the zero-tolerance gate compares live-stat vs index and would otherwise refuse. */ +function bumpRevision(root: string, docId: string, toMs: number): void { + const doc = getDoc(docId, root)!; + const file = path.join(root, "docs", docId, `${doc.title}.${doc.ext}`); + fs.utimesSync(file, new Date(toMs), new Date(toMs)); + const actual = fs.statSync(file).mtimeMs; // the value every later stat (and persistBytes) will see + const p = path.join(root, "index.json"); + const raw = JSON.parse(fs.readFileSync(p, "utf8")); + raw.docs.find((d: any) => d.docId === docId).mtimeMs = actual; + fs.writeFileSync(p, JSON.stringify(raw, null, 2)); +} + +/* ------------------------------------------------------------------ */ +/* registration + scope */ +/* ------------------------------------------------------------------ */ + +test("registers office_* only in agent and project home chats", () => { + const s = setup("chat-atlas-home"); + (setup as any).__run(); + assert.deepEqual( + s.pi.registered.map((t) => t.name).sort(), + ["office_comments", "office_create", "office_edit", "office_list", "office_read"], + ); + + // wrong kind (a plain chat folder): the factory RETURNS, nothing registered + const notes = makePi(); + registerTools({ + extend: (f) => f(notes, { id: "chat-notes" }), + chatFolder: (id) => CHAT_TO_FOLDER[id], + folders: () => FOLDERS, + inFolder: (p) => (HOME_BY_FOLDER[p] ?? []).map((id) => ({ id })), + root: s.root, + log: (l) => s.seen.logs.push(l), + }); + assert.equal(notes.registered.length, 0); + + // a chat whose folder is gone: no tools, one log line, no throw + const ghost = makePi(); + registerTools({ + extend: (f) => f(ghost, { id: "chat-gone" }), + chatFolder: () => { + throw new Error("folder gone"); + }, + folders: () => FOLDERS, + inFolder: () => [], + root: s.root, + log: (l) => s.seen.logs.push(l), + }); + assert.equal(ghost.registered.length, 0); + assert.ok(s.seen.logs.some((l) => l.includes("no workspace folder for chat chat-gone"))); +}); + +test("home chat of a multi-session project folder is the lowest id (ping.ts rule)", async () => { + const s = setup("zzz-acme-b"); // a NON-lowest session of the Acme project + const made = await tool(s, "office_create").execute("c", { title: "Acme Note", type: "word" }); + assert.equal(getDoc(made.details.id, s.root)!.owner, "aaa-acme-a"); // owner = the deterministic home, not the acting session +}); + +test("scope: owned and granted docs visible; another home's doc invisible to list AND read", async () => { + const s = setup(); + const home = "chat-atlas-home"; + await createDoc({ title: "Mine", type: "word", owner: home }, s.root); + const theirs = await createDoc({ title: "Theirs Secret Doc", type: "word", owner: "chat-other" }, s.root); // no grant to us + const granted = await createDoc({ title: "Shared One", type: "cell", owner: "chat-other" }, s.root); + addGrant(s.root, granted.docId, home, "c-grant"); + + (setup as any).__run(); + const listText = (await tool(s, "office_list").execute("c", {})) as ToolResult; + assert.ok(listText.content[0].text.includes('"Mine"')); + assert.ok(listText.content[0].text.includes('"Shared One"')); + assert.ok(!listText.content[0].text.includes("Theirs Secret Doc"), "another home's doc must not appear in office_list"); + + // office_read of the out-of-scope id refuses with the SAME words as a nonexistent id (docs convention: no leak) + const missing = await refused(tool(s, "office_read").execute("c", { id: "no-such-id" }), "missing id"); + const foreign = await refused(tool(s, "office_read").execute("c", { id: theirs.docId }), "out-of-scope id"); + assert.equal(foreign, missing); + assert.match(foreign, /^error:not_found /); + + // the granted doc IS readable (xlsx grid path per plan 5.5) + const shared = (await tool(s, "office_read").execute("c", { id: granted.docId })) as ToolResult; + assert.ok(shared.content[0].text.includes('"Shared One"')); +}); + +/* ------------------------------------------------------------------ */ +/* office_create */ +/* ------------------------------------------------------------------ */ + +test("office_create: happy path returns id + meta, owner is the home; duplicate title+type -> error:exists", async () => { + const s = setup(); + const t = tool(s, "office_create"); + const res = (await t.execute("c1", { title: "Budget", type: "cell" })) as ToolResult; + const meta = getDoc(res.details.id, s.root)!; + assert.ok(meta); // exists in the index and on disk + assert.equal(meta.owner, "chat-atlas-home"); + assert.equal(meta.ext, "xlsx"); + assert.equal(typeof res.details.mtimeMs, "number"); + assert.ok(res.content[0].text.includes(res.details.id)); + assert.ok(s.seen.records.some((r) => r.chatId === "chat-atlas-home" && r.reason.includes("office_create")), "audit recorded for create"); + + const dup = await refused(t.execute("c2", { title: "Budget", type: "cell" }), "duplicate"); + assert.match(dup, /^error:exists /); + + // a different type with the same title is NOT a duplicate + const word = (await t.execute("c3", { title: "Budget", type: "word" })) as ToolResult; + assert.ok(word.details.id && !word.content[0].text.startsWith("error:")); + + // bad type -> error:unsupported_format; blank title -> error:invalid (refused before the store is touched) + assert.match(await refused(t.execute("c4", { title: "Nope", type: "spreadsheet" }), "bad type"), /^error:unsupported_format /); + assert.match(await refused(t.execute("c5", { title: " ", type: "word" }), "blank title"), /^error:invalid /); +}); + +/* ------------------------------------------------------------------ */ +/* office_edit */ +/* ------------------------------------------------------------------ */ + +test("office_edit: applies ops to the stored bytes, bumps the revision; old baseRevision -> error:revision_mismatch with the fresh mtime; quote_not_found passes through", async () => { + const s = setup(); + const doc = await createDoc({ title: "Edit Me", type: "word", owner: "chat-atlas-home" }, s.root); + const base1 = getDoc(doc.docId, s.root)!.mtimeMs; + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "New document"); + + const t = tool(s, "office_edit"); + const ok = (await t.execute("e1", { id: doc.docId, ops: [{ quote: "New document", replacement: "Edited text" }], baseRevision: base1 })) as ToolResult; + assert.ok(!ok.content[0].text.startsWith("error:"), `unexpected error: ${ok.content[0].text}`); + assert.match(ok.content[0].text, /now at revision/); + + // round-trip: re-reading the STORED bytes shows the edit; persistBytes updated the index's revision + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "Edited text"); + assert.equal(getDoc(doc.docId, s.root)!.mtimeMs, ok.details.to); + + // a second edit on the OLD baseRevision is refused with the FRESH mtime in the message -- and nothing written + const fresh = getDoc(doc.docId, s.root)!.mtimeMs; + const stale = await refused(t.execute("e2", { id: doc.docId, ops: [{ quote: "Edited text", replacement: "again" }], baseRevision: fresh - 5000 }), "stale gate"); + assert.match(stale, /^error:revision_mismatch /); + assert.ok(stale.includes(String(fresh)), `the fresh mtime ${fresh} must be in the message: ${stale}`); + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "Edited text"); // unchanged + + // with the FRESH base it applies; a bad quote surfaces ooxml's quote_not_found code verbatim + const ok2 = await t.execute("e3", { id: doc.docId, ops: [{ quote: "Edited text", replacement: "Final words" }], baseRevision: getDoc(doc.docId, s.root)!.mtimeMs }); + assert.ok(!ok2.content[0].text.startsWith("error:"), ok2.content[0].text); + const qnf = await refused(t.execute("e4", { id: doc.docId, ops: [{ quote: "not in the document at all", replacement: "x" }], baseRevision: getDoc(doc.docId, s.root)!.mtimeMs }), "quote_not_found"); + assert.match(qnf, /^error:quote_not_found /); + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "Final words"); // still nothing written + + // spreadsheets cannot be edited in v1 (plan 5.5: document.xml run-level edits are Word-only) + const sheet = await createDoc({ title: "Grid", type: "cell", owner: "chat-atlas-home" }, s.root); + assert.match(await refused(t.execute("e5", { id: sheet.docId, ops: [{ quote: "a", replacement: "b" }], baseRevision: getDoc(sheet.docId, s.root)!.mtimeMs }), "xlsx edit"), /^error:unsupported_format /); + + // audit recorded for the two successful edits only + assert.equal(s.seen.records.filter((r) => r.reason.includes("office_edit")).length, 2); +}); + +test("office_edit co-editor model: a live human session neither blocks nor gates; a stale revision is refused with a loud CONFLICT", async () => { + const s = setup(); + const doc = await createDoc({ title: "Co Edit", type: "word", owner: "chat-atlas-home" }, s.root); + // Pin the file mtime to an exact value AND mirror it in the index so the ZERO-tolerance gate is deterministic + // (a fresh create can leave a sub-ms stat drift on some filesystems). + const REV1 = Date.now() - 5000; + bumpRevision(s.root, doc.docId, REV1); + setUser(doc.docId, { id: "human-1", name: "Ada" }, true); // a human has the editor open RIGHT NOW + + // Case A (INVERTED from v1's refuse-while-a-session-is-open model): matching revision + an open human + // session -> the edit SUCCEEDS. No settings.json write needed any more: there is no switch; presence never blocks. + const ok = (await tool(s, "office_edit").execute("f1", { id: doc.docId, ops: [{ quote: "New document", replacement: "Co-edited" }], baseRevision: getDoc(doc.docId, s.root)!.mtimeMs })) as ToolResult; + assert.ok(!ok.content[0].text.startsWith("error:"), ok.content[0].text); + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "Co-edited"); // the agent edit landed alongside the open session + + // Case B: an external co-editor save advanced the file mtime after the agent read it -> refused with a CONFLICT + // (code stays revision_mismatch), the FRESH mtime in the message, the open editor named so the agent learns a + // human is co-editing -- and NOTHING written. + const baseB = getDoc(doc.docId, s.root)!.mtimeMs; + bumpRevision(s.root, doc.docId, Date.now() + 5000); // guaranteed >1 ms ahead of base: an external save moved the revision + const freshAfterBump = getDoc(doc.docId, s.root)!.mtimeMs; // the value every gate now sees + const refusedMsg = await refused(tool(s, "office_edit").execute("f2", { id: doc.docId, ops: [{ quote: "Co-edited", replacement: "late" }], baseRevision: baseB }), "stale revision must refuse"); + assert.match(refusedMsg, /^error:revision_mismatch /); + assert.ok(refusedMsg.includes("CONFLICT"), `the refusal is an explicit CONFLICT: ${refusedMsg}`); + assert.ok(refusedMsg.includes(String(freshAfterBump)), `the fresh mtime ${freshAfterBump} must be in the message: ${refusedMsg}`); + assert.match(refusedMsg, /human editor \(Ada\)/, "the open session is named so the agent knows a human is co-editing"); + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "Co-edited"); // NOTHING written by the stale edit + + // Case C: same stale revision with NOBODY in the editor -> still refused (the gate is the safety mechanism, + // never a session check), but without the human-editor note. + resetLive(); + const refusedC = await refused(tool(s, "office_edit").execute("f3", { id: doc.docId, ops: [{ quote: "Co-edited", replacement: "still late" }], baseRevision: baseB }), "stale revision with no session must refuse too"); + assert.match(refusedC, /^error:revision_mismatch /); + assert.ok(!refusedC.includes("human editor"), "no open session -> no human-editor note"); + + // Case D: re-read first (fresh base) and the same edit succeeds again -- fail loud, re-read + retry is the loop. + const okD = (await tool(s, "office_edit").execute("f4", { id: doc.docId, ops: [{ quote: "Co-edited", replacement: "Retried after re-read" }], baseRevision: freshAfterBump })) as ToolResult; + assert.ok(!okD.content[0].text.startsWith("error:"), okD.content[0].text); + assert.equal(parseDocx(readDocBytes(s.root, doc.docId)).text, "Retried after re-read"); +}); + +/* ------------------------------------------------------------------ */ +/* office_comments */ +/* ------------------------------------------------------------------ */ + +test("office_comments: docx lists comment ids/authors/text + grants; xlsx -> unsupported note", async () => { + const s = setup(); + const doc = await createDoc({ title: "Commented", type: "word", owner: "chat-atlas-home" }, s.root); + // inject a comments part into the stored bytes (mtime moves -- the tool re-reads the meta anyway) + const file = path.join(s.root, "docs", doc.docId, `${doc.title}.${doc.ext}`); + fs.writeFileSync(file, rewriteZip(readDocBytes(s.root, doc.docId), { "word/comments.xml": commentsXml([{ id: "c1", author: "Ada Person", text: "@agent please fix this" }, { id: "c2", author: "Bob", text: "second note" }]) })); + addGrant(s.root, doc.docId, "chat-other", "c1"); + + const res = (await tool(s, "office_comments").execute("cm", { id: doc.docId })) as ToolResult; + const t = res.content[0].text; + assert.ok(!t.startsWith("error:"), t); + assert.match(t, /c1\s+Ada Person/); + assert.match(t, /@agent please fix this/); + assert.match(t, /c2\s+Bob/); + assert.match(t, /chat-other.*via comment c1/s); + assert.equal(res.details.comments.length, 2); + assert.equal(res.details.grants[0].chatId, "chat-other"); + + // a doc with no comments -> a readable "(none)", not an error + const bare = await createDoc({ title: "Bare", type: "word", owner: "chat-atlas-home" }, s.root); + const bareRes = (await tool(s, "office_comments").execute("cm2", { id: bare.docId })) as ToolResult; + assert.match(bareRes.content[0].text, /Comments \(0\):\n\(none\)/); + + // xlsx: the unsupported note with the docs-style code prefix (comments arrive in M4) + const sheet = await createDoc({ title: "Sheet", type: "cell", owner: "chat-atlas-home" }, s.root); + assert.match(await refused(tool(s, "office_comments").execute("cm3", { id: sheet.docId }), "xlsx comments"), /^error:unsupported_format /); +}); + +test("office_comments: aggregated comment output is capped like office_read (marker + bounded details)", async () => { + const s = setup(); + // 300 comments x ~750 chars of text -- comfortably over the shared 200,000-character cap + const many = Array.from({ length: 300 }, (_, i) => ({ id: String(i), author: "Crowd", text: `note ${i}: ${"x".repeat(750)}` })); + const doc = await createDoc({ title: "Wall of Notes", type: "word", owner: "chat-atlas-home" }, s.root); + const file = path.join(s.root, "docs", doc.docId, `${doc.title}.${doc.ext}`); + fs.writeFileSync(file, rewriteZip(readDocBytes(s.root, doc.docId), { "word/comments.xml": commentsXml(many) })); + + const res = (await tool(s, "office_comments").execute("cm4", { id: doc.docId })) as ToolResult; + const t = res.content[0].text; + assert.ok(!t.startsWith("error:"), t.slice(0, 200)); + assert.match(t, /Comments \(300\):/, "the true total is announced even when truncated"); + assert.ok(t.length <= 200_000 + 100, `output bounded (got ${t.length} chars)`); + assert.match(t, /\[output truncated by office_comments: it exceeds 200,000 characters\]/); + + // details are bounded exactly like office_read's: 100 at 500 chars each, true total in commentCount + assert.equal(res.details.commentCount, 300); + assert.equal(res.details.comments.length, 100); + for (const c of res.details.comments) assert.ok(c.text.length <= 500); +}); + +/* ------------------------------------------------------------------ */ +/* error string format (docs convention, office fix) */ +/* ------------------------------------------------------------------ */ + +test("every refusal the agent reads starts with \"error:<code> \"", async () => { + const s = setup(); + const doc = await createDoc({ title: "Fmt", type: "word", owner: "chat-atlas-home" }, s.root); + const sheet = await createDoc({ title: "F2", type: "cell", owner: "chat-atlas-home" }, s.root); + + assert.match(await refused(tool(s, "office_read").execute("x1", { id: "nope" }), "not_found"), /^error:not_found /); + const stale = await refused(tool(s, "office_edit").execute("x2", { id: doc.docId, ops: [{ quote: "New document", replacement: "r" }], baseRevision: getDoc(doc.docId, s.root)!.mtimeMs - 9000 }), "stale gate"); + assert.match(stale, /^error:revision_mismatch /); + assert.match(await refused(tool(s, "office_edit").execute("x3", { id: doc.docId, ops: [], baseRevision: getDoc(doc.docId, s.root)!.mtimeMs }), "empty ops"), /^error:invalid /); + assert.match(await refused(tool(s, "office_create").execute("x4", { title: "Fmt", type: "word" }), "exists"), /^error:exists /); + assert.match(await refused(tool(s, "office_comments").execute("x5", { id: sheet.docId }), "xlsx comments"), /^error:unsupported_format /); +}); + +/* ------------------------------------------------------------------ */ +/* the wake drain side (plan 5.4 step 4) */ +/* ------------------------------------------------------------------ */ + +test("the factory also registers the UI_RECEIVE ping drain for office chats only", () => { + const s = setup(); + (setup as any).__run(); + assert.equal(s.pi.uiHandlers.length, 1); + // a non-ping bell is ignored without draining anything or throwing + s.pi.uiHandlers[0]({ data: { type: "other" } }); + assert.equal(s.pi.sent.length, 0); + + const notes = makePi(); + registerTools({ + extend: (f) => f(notes, { id: "chat-notes" }), + chatFolder: () => "/f/notes", + folders: () => FOLDERS, + inFolder: () => [], + root: s.root, + log: (l) => s.seen.logs.push(l), + }); + assert.equal(notes.uiHandlers.length, 0); // no tools in a plain chat -> no drain either +}); diff --git a/extensions/office/tsconfig.json b/extensions/office/tsconfig.json new file mode 100644 index 000000000..01bdfe196 --- /dev/null +++ b/extensions/office/tsconfig.json @@ -0,0 +1,14 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ESNext", + "moduleResolution": "Bundler", + "jsx": "react-jsx", + "lib": ["ES2023", "DOM", "DOM.Iterable"], + "types": ["node"], + "strict": true, + "noEmit": true, + "skipLibCheck": true + }, + "include": ["src"] +} diff --git a/tasks/pr307-callback-concurrency.md b/tasks/pr307-callback-concurrency.md new file mode 100644 index 000000000..ffbf1312c --- /dev/null +++ b/tasks/pr307-callback-concurrency.md @@ -0,0 +1,84 @@ +# PR #307 review — callback loop, mtime revision gate, store & container lifecycle (concurrency/correctness) + +Repo: `/root/repos/pithagoras`, branch `office/280-pithagoras-office` @ `fee666ce`, diff vs `main`. +Slice files read in full: `extensions/office/src/callback.ts` (521), `store.ts` (459), `container.ts` (336), `ping.ts` (273); skimmed `live.ts`, `jwt.ts`, relevant parts of `server.ts` and `tools.ts`; tests skimmed: `test/callback.test.mts`, `test/store.test.mts`, `test/container.test.mts`, `test/ping.test.mts`. + +## Scope +1. DS callback status-table fidelity (incl. the 200-with-`{"error":1}` trick, retry behavior). +2. mtime revision gate: TOCTOU/atomicity, index atomicity, mtime granularity, clock changes. +3. Agent-edit path: forcesave-first ordering vs the expectedMtimeMs gate; concurrent agent edit + DS persist. +4. Duplicate / out-of-order callbacks and idempotency. +5. Trash sweep at the 7-day boundary; concurrent index writers. +6. Container lifecycle: adopt-never-recreate, ds.json origin drift, volume reuse, loopback port allocation, netns handling. +7. lookAfter/ping health loops. + +## Method +Full read of the four slice modules and both adjacent seams (tools edit path, server boot/loopback); read all four matching test files to map already-covered races; cross-checked the handler against the official DS callback contract (api.onlyoffice.com callback-handler page: statuses 1/2/3/4/6/7 in the current table, `users` on status 1, `url` on 2/3/6/7, `forcesavetype` on 6/7, editor shows an error when the answer is not `{error:0}`); ran a throwaway fs probe in /tmp (deleted afterwards): 20k consecutive writes show sub-ms `mtimeMs` deltas (min 7.8 µs) and that rename preserves the tmp file's write time as the new revision mtime. Checklists: `cross-cutting/async-concurrency-patterns.md`, `cross-cutting/error-handling-principles.md`. + +## Findings + +### 🔴 [blocking] +(none in this slice — the status table, envelope binding, and persist gate are all correct against the current DS contract; details under "What's good".) + +### 🟡 [important] + +**1. Latent infinite stall: a *working* forcesave hook makes every `office_edit` with an open human session fail.** +- Where: `src/tools.ts:373–379`. After `const flushed = await forcesaveHook(doc.docId)` succeeds, the re-gate compares against the ORIGINAL base: `if (Math.abs(meta.mtimeMs - base) > 1) throw stale(meta);` (line 377). +- Why it matters: a completed force-save is exactly what moves the file mtime by far more than 1 ms — so the re-check throws `revision_mismatch` for every future forcesave flow, and the agent loops forever ("changed since you read it") while the human has the document open. The comment at line ~372 says the re-check exists because "that save moves the file" — but compares against the wrong baseline. Not reachable today only because `makeForcesaveHook()` (`src/server.ts:140–148`) always returns `false` while users are present (documented M0 deferral); when the real hook lands, this line defeats it. +- Fix: capture `const preFlushMtime = getDoc(...)?.mtimeMs` before calling the hook; after a successful flush accept the fresh meta if `meta.mtimeMs >= preFlushMtime` (the advance IS our forcesave persist — or re-check disk-vs-meta strictly, which `persistBytes` does anyway). Reject only if the revision moved again *between* flush completion and now. + +**2. `persistBytes` can strand a document: file renamed, index update failed/crashed, no reconciliation path.** +- Where: `src/store.ts:449–457`. Order is: gate (440–446) → write tmp + rename (449) → re-stat (450) → `readIndex`/mutate/`writeIndex` (452–457). If the index write throws (ENOSPC, EIO, corrupt-index surprise) or the process dies between line 449 and 457, the on-disk file has a NEWER mtime than `index.json` records. +- Why it matters: from then on every save for that doc fails — callback stale gate passes (key matches old index mtime) but `persistBytes`' strict check (`st.mtimeMs !== doc.mtimeMs`, line 444) throws `revision_mismatch`; `office_edit`'s first gate also refuses. The document becomes permanently unsaveable until someone hand-edits the index. No self-heal exists: `settleKept`/`pruneMissingDocs` (`src/store.ts:336–351`) only drops entries whose *file* is missing — a disk-newer-than-index entry is invisible to it, and the "run settleKept to reconcile" hint on `index_corrupt` (line 57) does not cover this case. +- Fix (concrete): keep the ordering but add an explicit resync escape hatch — e.g. store ops detect `diskMtimeMs > meta.mtimeMs` and log a distinct, greppable line (`revision diverged: disk ahead of index for <docId> — run repair`) plus an admin op (tool or route) that re-baselines the entry's `mtimeMs`/`sizeBytes` from disk after human confirmation. Alert on the existing `persist for <docId>: store failure:` log line (`src/callback.ts:478–479`). +- Confidence: high (ordering verified in code; trigger requires a failed index write or crash in a window of a few ms — realistic under ENOSPC on a box running a document server). + +### 🟢 [nit] / 💡 [suggestion] + +**3. 🟢 Revision identity is mtime-only; coarse-granularity filesystems can merge two revisions.** +- Where: `src/callback.ts:104–108` (`revHashOf` = sha256 of `String(mtimeMs)`), gate at `src/store.ts:444`, key mint at `src/server.ts:~256` (`callback.makeKey(doc.docId, doc.mtimeMs)`). +- Why: on an fs with 1–2 s mtime granularity (CIFS/SMB, HFS+, FAT), two legitimate saves landing in the same bucket produce identical `mtimeMs` → identical revHash → both the stale gate and the strict persist check treat a stale session as current, so the older revision silently overwrites the newer one. On this host I measured min consecutive-write deltas of 7.8 µs, so ext4-class storage is fine; this is a deployment-shape hazard (and `String(mtimeMs)` float formatting keeps sub-ms distinctions only while the fs reports them). +- Fix: add a monotonically-increasing per-doc counter to `DocMeta`, embed it in the key (`office|<docId>|<n>-<hash>`), and compare it in both gates; keep mtime as the human-facing revision. Cheap, kills the whole class. + +**4. 🟢 Tolerance inconsistency: tools use ±1 ms, everything else is zero-tolerance.** +- Where: `src/tools.ts:365` and `:377` (`Math.abs(meta.mtimeMs - base) > 1`) vs strict `!==` at `src/store.ts:444` and hash equality in `persistSaved`. +- Why: the ±1 ms window matters only for drift ≤1 ms (backward NTP step, two revisions <1 ms apart — the fs probe shows sub-ms deltas are real on ext4). No clobber is possible because the final `persistBytes` gate compares disk-vs-index regardless of base; but the design summary claims "zero tolerance" and a reader can't tell which invariant holds where. Align both to `meta.mtimeMs !== base` (or document the epsilon deliberately). + +**5. 🟢 [speculative, low confidence] Rename/delete landing inside the callback's fetch window loses one human save.** +- Where: `src/callback.ts:430` (`store.getDoc`) → `await fetchWithRetry(...)` (~350 ms worst case) → `persistBytes(meta, ...)` at `src/callback.ts:474` with the title-path from the pre-fetch meta; path built at `src/store.ts:~78` (`docs/<docId>/<title>.<ext>`). +- Why: a human retitle in that window makes the file path ENOENT → logged as generic "store failure" (`src/callback.ts:478–479`) → save refused, the human's changes gone for this session; a delete behaves the same (refused, but the doc is gone anyway). Narrow timing window, single one-time loss. Fix: re-resolve the meta by `docId` immediately before the persist in `persistSaved`, and map ENOENT to its own log line. + +**6. 💡 Ephemeral loopback port doesn't survive a portal restart, but open DS sessions do.** +- Where: `src/server.ts:528` (`server.listen(0, "127.0.0.1", ...)`), config mint at `~244` (`base = http://127.0.0.1:${loopPort}`), cleanup sets `loopPort = 0`. +- Why: the DS container is adopt-never-recreate and outlives portal restarts; every in-flight editor session's `callbackUrl` then points at the dead port — next save fails (editor error; no clobber, since the key pins the old revision) until the user reopens the editor. No retry-storm risk: per the DS contract a non-`{error:0}` answer only shows an editor message, and our handler always answers every persist failure with 200 `{"error":1}`, and internal retries are bounded to 3 attempts (`src/callback.ts:~205`). Fix: pick a stable high port once (persist in settings or `ds.json`, retry next on EADDRINUSE) so sessions survive restarts; also guard `buildEditorConfig` against `loopPort === 0`. + +**7. 🟢 ds.json origin is never refreshed while the DS keeps running.** +- Where: `src/container.ts:330` (`lookAfter`: `if (existing?.state === "running") return;` skips `ensureContainer`), vs the rewrite at `src/container.ts:224`. +- Why: a deployment-shape flip (host netns ↔ portal-container netns) with a running DS leaves the recorded `origin` stale until the next stop/start — `api.js` src (`src/server.ts:418`) and the M0-marker logic then use the wrong origin. Rare, self-limiting; fix by validating/rewriting origin during `lookAfter` (or on every `ensureContainer` success it already does — just reach that path when only the origin is stale). + +**8. 🟢 Single-process assumption for index.json is documented but unenforced.** +- Where: all index writers (`src/store.ts:63–71` writeIndex, `src/ping.ts:158+` addGrant raw read-modify-write) rely on the Node event loop: each writer's read→mutate→rename block contains no await, so it is atomic within one process (verified — that's why the code is deliberately sync). Two portal processes over one dataDir would silently clobber each other's entries (no O_EXCL/lockfile), and `writeIndex` tmp names (`.<pid>.tmp`) only serialize same-pid. Fix: an advisory pidfile lock on dataDir with a loud error, or at least state it in the README next to the "deferred" list. Related nit: `addGrant` re-implements a raw index writer instead of going through store.ts exports (the module comment admits this) — drift risk; fold into a store grant-mutation op. + +**9. 💡 Synchronous full-document parsing on the persist hot path.** +- Where: `src/ping.ts:~230` (`parseDocx(buffer)` inside `handlePersisted`, fired synchronously by the callback hook loop at `src/callback.ts:500–507`); same pattern in `store.importFile` (`fs.readFileSync` + `parseDocx`, `src/store.ts:383–390`). +- Why: multi-MB docx parsed with the sync zlib codec blocks the event loop for tens of ms per persist — it also delays answering DS, stretching the window where the editor shows "saving". Acceptable at current scale; if docs grow large, move diff/wake to a queue processed after the `{error:0}` answer (the hook already runs post-answer-prepare, so ordering is safe to change). + +**10. 🟢 Unbounded in-memory wake queue.** +- Where: `src/ping.ts:~83` (`const wakes = new Map<string, WakeEntry[]>()`, drained only by the target chat's UI_RECEIVE handler in `tools.hearOfficePings`). If a home chat never registers the drain (no session, older pi build without event bus — the guard at `src/tools.ts:~470` returns silently), entries accumulate for the process lifetime. Tiny objects, but add a size cap + log line on overflow. + +### What's good +- **Status-table fidelity is correct against the current (10.x) DS contract** — I verified against the official callback-handler docs: statuses 1 (users[] join/leave snapshot), 2 (saved, url+changesurl, persist-then-unlock), 3/7 (DS-side error → log only, answer `{error:0}` since we didn't fail), 4 (no changes → clear presence AND lock), 6 (force-saved, stays locked) are all handled exactly, and `forcesavetype` is correctly treated as 6/7-only. There is no status 0 in the current table (older docs' pre-connection status was folded into status 1 with `actions[]`). The legacy-shape `actions[].type === 2` persist on a status-1 envelope is harmless defensive code (`src/callback.ts:~340`). +- **Envelope binding from signed JWT claims** (`src/callback.ts:~285–300`) is the right call: the unsigned body is ignored entirely, a valid non-callback token cannot drive a hostile POST, and `verifyJwt` pins `alg:"HS256"` with `timingSafeEqual` and exp checks (`src/jwt.ts:66–84`). Token verified exactly once per request (documented rationale in the code). +- **The persist path is event-loop atomic where it must be**: `persistBytes` has no await between stat-gate, tmp write, and rename (`src/store.ts:440–457`) — concurrent DS-persist + agent-edit on one doc serialize to a clean last-writer-wins with the loser getting `revision_mismatch`, which is exactly the intended clobber protection. Same for every index read-modify-write block. +- **Duplicate force-save idempotency** (`lastPersist` memo, `src/callback.ts:~140–165`) handles the documented forcesave→status-6 double pass, is scoped to (url, our revision), fails closed on any newer change at a different url, and has a direct test (`duplicate force-save pass` in `test/callback.test.mts`). +- **No retry storms**: every persist failure answers 200 with `{"error":1}` (correct — DS reads the JSON field), internal download retries are bounded to 3 attempts over ~350 ms, fetch-failure leaves lock state untouched for DS-side visibility, and 2 MB body cap with drain. +- **`ensureContainer` serializes the whole lifecycle** (`src/container.ts:~157–165`, cleared in `finally`), so the 30 s `lookAfter` interval joins the in-flight pull instead of piling up; user-stopped marker is written BEFORE the stop to close the lookAfter race; tag changes only recreate *stopped* containers (no surprise restarts, named volume preserves DS data). +- **Trash handling** is race-free within its stated model: stamps carry the docId so same-millisecond deletions of different docs never collide, `sweepTrash` skips non-stamped entries and runs on a 30 s timer that can't preempt the sync delete/sweep blocks, and there is no in-app restore to race against. +- **`live.ts` lock model** is conservative in the safe direction (presence OR flag; terminal closes clear both — so a server-minted seat without a DS leave event won't badge forever), with the mint-time `setUser` in `buildEditorConfig` covering session detection even before any DS callback arrives. + +### Open questions +1. **Forcesave path**: when M0 wires the real tier-1 flush, do you go via command-service or the editor UI `forcesave:true` option? Finding 1 must be fixed whichever way; also decide whether the status-6 that then arrives needs to clear `lastPersist` semantics (today it's absorbed by the memo). +2. **"Undoable" trash**: the PR copy says "7-day undoable trash", but there is no restore op or route anywhere in `src/` (I grepped — only `deleteDoc`→`.trash`, then `sweepTrash`). Manual `mv` works but is undocumented. Is an in-app restore deferred, or should the README wording be corrected? +3. **Revision identity**: will you take finding 3's rev-counter suggestion before anyone runs this on SMB/NFS-backed storage? It's a one-field change that makes the whole gate bulletproof. +4. **Loopback port stability** (finding 6): fixed persisted port vs accept "re-open the editor after a portal restart" as documented behavior? +5. **Repair semantics** (finding 2): should re-baselining disk→index be an operator op, or is an auto-heal at boot acceptable once you've logged a divergence? diff --git a/tasks/pr307-ci-failure.md b/tasks/pr307-ci-failure.md new file mode 100644 index 000000000..2053d201a --- /dev/null +++ b/tasks/pr307-ci-failure.md @@ -0,0 +1,95 @@ +# PR #307 CI failure — `extensions/docs` rename-recovery test + +**Task:** explain the single failing test of CI run 40036094127 on branch `office/280-pithagoras-office` (`fee666ce`): + +``` +not ok 366 - a rename that was cut off between its halves is finished only for the file it was decided for: + a file with the two names that is another one is left as it is +location: extensions/docs/test/rename.test.mts (line 759) +error: 'neither name of the other file was taken away' (deepStrictEqual; expected ['after.md','plan.md'], actual ['after.md']) +``` + +## Scope + +- Branch `office/280-pithagoras-office` at `fee666ce`, base = local `main` = merge-base `138738f5` (branch is on current main; PR adds `extensions/office`, +8703 lines, and touches exactly one core test line — it **does not touch `extensions/docs` at all**). +- The failing file: `extensions/docs/test/rename.test.mts:740` (subtest), under test: `resume("rename")` / `#carryRename` in `extensions/docs/store.js`, descriptor handling in `core/server/src/workspace-files.ts`. Area introduced by `fc6d396c` ("Documents on portal.files: … descriptor-based file handling"). +- CI model (GitHub Actions, job `server`, ubuntu-latest): one `node --test` invocation over all files; each test **file** runs in its own child process, and the 32 subtests of this file run **sequentially in one process**. "not ok 366" is the global TAP number across the whole suite, not a per-file index. + +## Method + +- Read the subtest (`rename.test.mts:740–771`) and the implementation it covers (`store.js` `recover()` at :501–510, `#carryRename` at :1086–1137, `sameId` at :91, `files.js removeDocument` :140–143, `workspace-files.ts removeFile` :1222). +- Reproduced on the branch and on a clean worktree of the merge-base (no PR code): 5 full-file runs each, plus an isolated-subtest loop (`--test-name-pattern`). +- Instrumented a scratch copy of the merge-base (`/tmp/pithagoras-trace`, removed after use; per-run logs kept in `/tmp/tr`) with stderr traces of every `recover()`, `#carryRename` entry (full `old`/`now`/pending stats), pending write, and step execution. +- Ran an inode-reuse micro-benchmark on this machine's ext4, and a 100-run full-file campaign **with concurrent file-system churn** (a filler process allocating/freeing files) so the allocator would hand the original inode back to the recreated file. + +Reproduction tally (one full-file run = all 32 subtests): + +| Setting | Runs | Result | +|---|---|---| +| Branch, quiet host | 5 | 0 failures | +| Merge-base worktree (no PR code), quiet host | 5 | **1 failure — this subtest, exactly CI's assertion** | +| Instrumented merge-base copy, quiet host | ~64 | 0 failures; the recreated file got `origIno − 11` every time (ext4 "inode goal" slot), so the bug branch never armed | +| Isolated subtest (`--test-name-pattern`), quiet host | 90 | 0 failures — without allocator luck the subtest cannot trigger | +| Instrumented merge-base copy, **with fs churn** | 100 | **21 failures of this subtest**, every one with proven inode reuse; plus 6 failures of a sibling deletion subtest (Finding 5); run 97's 18 failures are an infra artifact (the scratch tree was removed mid-run by something outside this investigation — `git: fatal: Unable to read current working directory`) | + +Micro-benchmark (ext4, `/dev/mapper/pve-vm--122--disk--0`): create→unlink→create re-used the same inode in **~99%** of 3000 trials on a quiet host; churn varies which freed slot comes back — that is what arms the bug branch. + +## Findings + +1. 🔴 **The failure reproduces on `main` without any PR code — pre-existing, not PR-caused.** Full-file run on the merge-base worktree (`138738f5`, no `extensions/office`): failed exactly this subtest with exactly CI's assertion (expected `['after.md','plan.md']`, actual `['after.md']`, "neither name of the other file was taken away"). + +2. 🟡 **The only code path that can produce the observed disk state is `#carryRename`'s recovery branch at `extensions/docs/store.js:1108–1109`.** After the test recreates both names as a *new* hard-linked pair, the second portal's `GET /folders` → `Store.open`/`recover()` → `resume("rename")`. The branches of `#carryRename`: + - :1107 `old?.isFile() && !now` — impossible (`after.md` exists); + - :1108 `recovering && old.isFile && now.isFile && same ino && nlink > 1 && sameId(old, file)` → `removeDocument(dir, 'plan.md', { shared: true })` — **unlinks the name `plan.md`**. This is the *only* statement anywhere that can remove `plan.md` here (`takeIn` never unlinks; `sweepTemp` only matches `.name.hex.tmp` at `workspace-files.ts:1494`; delete-pending is not armed in this subtest; commit/gc act only in the store's own git dir under `DATA/<folderId>`). Trace run 7 shows exactly this branch's stat signature followed by the file gone (Finding 4). + - :1112 otherwise → Refused "taken" → dropped, disk untouched (the designed behavior the test asserts). + If :1108 fires, `#carryRename` then commits a rename revision and clears pending — leaving exactly `['after.md']`, i.e. **CI's exact observable state**. + +3. 🔴 **Root cause (data-loss class): file identity is decided from `{ dev, ino }` alone, and inode numbers are reused after a file is deleted.** + - The decision stores `file: idOf(was)` where `idOf = { dev: String, ino: String }` (`store.js:88`, captured at :1007/:1014) — no size, no mtime. `sameId` (:91) compares only those two strings. The window between capture and recovery spans the whole crash/stop gap plus **arbitrary activity by other actors**, so a *different* file made at either name can legitimately hold the same `{ dev, ino }`. + - On ext4 this is not hypothetical: ~99% immediate reuse in the micro-benchmark; under churn it happened in 21 of 96 subtest attempts. The test's own recreation (`rename.test.mts:752–755`: unlink both names → `writeFileSync` new `plan.md` → `linkSync` to `after.md`) lands back on the original inode whenever the allocator hands it out; then :1108 sees "same dev+ino, nlink 2", mistakes the agent's file for the store's own half-finished rename, unlinks `plan.md`, and commits a bogus rename revision. + - The design *knows* this danger: `workspace-files.ts`'s `linkOver` comment explicitly says "a folder swapped onto the name can have it (ext4 does)" and defends with a same-call re-stat + nlink check; `#carryRename`'s recovery branch has no equivalent defense because its reference stat is from the decision time, not milliseconds ago. + - The other use of `sameId` (`store.js:1164`, `#carryDelete`) matches in the opposite branch direction — a match *authorizes* deletion — so inode reuse has the same harmful effect there (Finding 5). + +4. 🔴 **Direct trace proof of the mechanism (churn-campaign run 7; runs 11 and 12 identical):** + + ``` + T-origIno ino=697244 ← original plan.md's inode O + setPending {op:rename, from:plan.md, to:after.md, file:{dev:64522,ino:697244}} ← the decision + T-recreated ino=697244 reused=true listing=[.keep, after.md, plan.md] ← ext4 hands O back to the agent's new hard-linked pair + carryRename recovering=true file{ino:697244} old{ino:697244,nlink:2} now{ino:697244,nlink:2} → stepRun + ← store.js:1108 conditions all true; the step is removeDocument(dir,'plan.md',{shared:true}) + T-afterGet listing=[.keep, after.md] ← plan.md taken away; CI's exact state + ``` + + In every one of the 21 red runs `T-origIno == T-recreated` (inode reuse); in every green run they differ and `plan.md` survives. A counter-example run with different inodes behaved correctly — the file was left alone. + +5. 🟡 **A sibling test fails through the same rule: `#carryDelete` (`store.js:1164–1183`).** Its guard is `if (there?.isFile() && !sameId(there, file)) { /* leave it */ } else { removeDocument(...) }`. When the file system re-assigns the decided inode to another actor's file at the name, `sameId` is true and recovery deletes that other file. Subtest `a deletion that is not known to have removed the file does not remove another file that is at the name, whatever it is called by then` (`rename.test.mts:866`) failed 6× in the churn campaign with `ENOENT … documents/plan.md` — its recreated file was taken away. Same root cause (dev+ino identity re-validated across a stop); the fix below covers both paths. + +6. 🟢 **Why it looks like a rare flake:** whether the recreation re-gets the original inode depends on the file system's allocator state (ext4 allocates with an "inode goal" near the creating directory; free slots elsewhere in the group are taken first). On a quiet host the recreated file consistently got `origIno − 11`, so the branch never armed (0/64 + 0/90); under churn it armed in 21/96 attempts (~22%) — matching the 1/5 seen on the plain merge-base worktree. A shared CI runner is exactly the noisy allocator state. **The tests are deterministic in their *intent* but allocator-dependent in their power to catch the bug.** + +## Verdict + +**Not PR-caused. A pre-existing latent bug in `extensions/docs` rename/deletion recovery, surfaced as a flaky CI failure by inode reuse after a deleted file's number is re-allocated.** + +- The PR could not cause it: it changes nothing in `extensions/docs` or the shared `core/server/src/workspace-files.ts`, and each test file runs in its own process (`node --test`) with per-run scratch dirs — there is no cross-file, cross-extension state. And the identical failure was reproduced on a worktree at the merge-base containing none of the PR's commits. +- Severity: 🟡 leaning 🔴 — the store can silently take away a name of a file that belongs to another actor (data loss), but the trigger needs an interrupted rename *or* deletion **and** both names being re-made by someone else on the same reused inode as a hard-linked pair; rare in practice. +- For PR #307: this failure is unrelated; a rerun is the right short-term action, and the underlying issue should be filed separately against `extensions/docs`. + +### Suggested fix (store.js) + +Treat "same file" as identity-plus-unchanged, not dev+ino alone: + +1. Widen the decision's fingerprint: at `rename()`/`delete()` capture `{ dev, ino, size, mtimeNs }` (all present on the `statOf` result; `idOf` :88 already receives it) and store it as `file`. +2. Require *all four* to match at `#carryRename` :1108 **and** at `#carryDelete` :1164 (`sameId` extended accordingly). A real half-finished rename or delete left the file untouched (content/mtime unchanged), so recovery still completes; a reused-inode impostor has a fresh mtime (and usually a different size) and is left alone. Residual TOCTOU (an attacker rewriting exactly the original bytes + mtime) is out of scope for this layer, as with `linkOver`. +3. Optionally also compare `blobId(readBytes(from)) === row.blob` in :1108 before unlinking (the store already reads the text a few lines later) — content-identical impostors would still pass dev+ino+mtime on some fs clocks; this is the strongest practical check available by name. + +### Suggested test hardening + +- Keep `rename.test.mts:740` as is — it documents the intended behavior and *does* catch the bug when the allocator cooperates (which it did in CI). +- With fix (1) above the recreation in the test has a different size **and** mtime from the original, so both affected subtests become deterministic with respect to the fixed implementation regardless of inode allocation. +- Add one more subtest that cannot depend on allocation luck: drive `#carryRename` recovery against a stubbed `files` where `statOf`/identity says "same dev+ino" but `readBytes` returns different content, and assert the old name is kept (content check of fix 3). A matching deletion-recovery subtest should pin the same invariant for `#carryDelete`. + +## Open questions + +- The one open loose end: run 97 of the churn campaign lost its scratch tree mid-run to something outside this investigation (an external `/tmp` cleaner on this host — it also removed both working copies after the loop finished); that run's failures are infra artifacts and excluded above. If that cleaner recurs, future reproduction campaigns should keep code outside `/tmp`. +- Whether `overlayfs` upper-layer behavior on CI runners changes reuse odds vs plain ext4 (direction unclear; the CI failure proves a matching dev+ino with nlink 2 occurred there). diff --git a/tasks/pr307-coderabbit-comments.md b/tasks/pr307-coderabbit-comments.md new file mode 100644 index 000000000..93cddf7d0 --- /dev/null +++ b/tasks/pr307-coderabbit-comments.md @@ -0,0 +1,220 @@ +# PR #307 issue comments (fetched 2026-10-23T23:46:47.217Z) + +--- +## commenter: coderabbitai[bot] | created: 2026-10-23T19:52:38Z +<!-- coderabbit-ai-generated:v1 --> + +<!-- This is an auto-generated comment: summarize by coderabbit.ai --> +<!-- review_stack_entry_start --> + +<a href="https://app.coderabbit.ai/change-stack/thecodacus/pithagoras/pull/307?scope=ghh_OusAJGA4HG8ljoOV0wJvVx8NmnZIjc2DdjmxOu0xZWA&cs_source=review_comment"><picture><source media="(prefers-color-scheme: dark)" srcset="https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui-dark.svg?v=3"><source media="(prefers-color-scheme: light)" srcset="https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg?v=3"><img src="https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg?v=3" alt="Review in Change Stack →" height="32"></picture></a> + +<!-- review_stack_entry_end --> +<!-- walkthrough_start --> + +<details> +<summary><strong>📝 Walkthrough</strong></summary> +<dl> +<dd> + +This pull request adds an Office extension for document storage, browser editing through OnlyOffice, chat-scoped document tools, and comment-triggered wake messages. It also adds settings and container lifecycle management, authenticated document and callback routes, OOXML processing, and tests. + +**Changes: Office extension** + +|Layer / File(s)|Summary| +|:---|:---| +|**Extension package and build setup** <br> `core/server/test/extension-ui.test.mjs`, `extensions/office/.gitignore`, `extensions/office/README.md`, `extensions/office/build.mjs`, `extensions/office/deps.js`, `extensions/office/package.json`, `extensions/office/scripts/make-fixtures.mjs`, `extensions/office/tsconfig.json`|Adds the Office package manifest, dependency and build scripts, TypeScript configuration, generated-output ignores, fixture-generation script, and extension documentation. The shipped-extension test includes `pithagoras-understory`.| +|**OOXML processing and document library** <br> `extensions/office/src/ooxml.ts`, `extensions/office/src/store.ts`, `extensions/office/test/fixtures.test.mts`, `extensions/office/test/ooxml.test.mts`, `extensions/office/test/store.test.mts`|Adds ZIP, DOCX, and XLSX parsing and writing; indexed document creation, import, export, trash cleanup, and revision-checked persistence; and tests for these operations.| +|**Settings and Document Server lifecycle** <br> `extensions/office/src/host.ts`, `extensions/office/src/settings.ts`, `extensions/office/src/container.ts`, `extensions/office/test/container.test.mts`, `extensions/office/test/settings.test.mts`|Adds settings persistence and portal and Docker access. The container module creates or adopts the configured Document Server container, reports its status, and records explicit stop intent.| +|**Editor access, callbacks, and server routes** <br> `extensions/office/src/jwt.ts`, `extensions/office/src/tickets.ts`, `extensions/office/src/callback.ts`, `extensions/office/src/server.ts`, `extensions/office/test/callback.test.mts`, `extensions/office/test/server.test.mts`, `extensions/office/test/integration.test.mts`|Adds signed editor configuration, ticketed document delivery, verified callbacks, server routes, and server lifecycle handling. Callback persistence checks document revisions before writing downloaded bytes.| +|**Chat tools, live state, and comment wakes** <br> `extensions/office/src/live.ts`, `extensions/office/src/tools.ts`, `extensions/office/src/ping.ts`, `extensions/office/test/tools.test.mts`, `extensions/office/test/ping.test.mts`, `extensions/office/test/integration.test.mts`|Adds chat-scoped document tools, presence and lock state, co-edit force-save checks, and comment-mention processing that records grants and queues wake messages.| +|**Browser library, editor, and settings** <br> `extensions/office/src/client.tsx`, `extensions/office/src/portal.tsx`, `extensions/office/src/office-ui.ts`, `extensions/office/src/web/*`, `extensions/office/style.css`, `extensions/office/test/office-ui.test.mts`|Adds Office navigation and chat UI, shared browser state and API calls, a document library and editor page, settings controls, and related styles.| + +<!-- change_assessment_start --> +**Priority:** ➖ Normal + +**Estimated code review effort:** 5 (Critical) | ~120 minutes + +<!-- change_assessment_commit:"fee666ce7b7b846a6d941992d0755f330331c83c" --> +**Change:** Feature +<!-- change_assessment_end --> + +### Sequence Diagram(s) + +```mermaid +sequenceDiagram + participant OfficeEditor + participant officeServer + participant DocumentServer + participant LoopbackListener + participant CallbackHandler + participant DocumentStore + OfficeEditor->>officeServer: Request editor configuration + officeServer-->>OfficeEditor: Return config with ticketed document and callback URLs + DocumentServer->>LoopbackListener: GET document using ticket + LoopbackListener->>DocumentStore: Read document after ticket verification + DocumentServer->>LoopbackListener: POST signed callback + LoopbackListener->>CallbackHandler: Forward callback request + CallbackHandler->>DocumentStore: Persist downloaded bytes with revision check +``` + +**Possibly related issues** + +- [thecodacus/pithagoras#280](https://github.com/thecodacus/pithagoras/issues/280) — The issue proposes the Office extension’s document tools, OnlyOffice integration, comment wakes, callback authentication, and Docker lifecycle. This pull request implements those areas. + +**Possibly related PRs** + +- [thecodacus/pithagoras#294](https://github.com/thecodacus/pithagoras/pull/294) — This PR changes how remembered server-half tools appear when their package is switched off or uninstalled; the Office extension registers its tools through that portal path. +- [thecodacus/pithagoras#288](https://github.com/thecodacus/pithagoras/pull/288) — This PR adds the server audit contract used by the Office document actions and comment wakes. + +**Suggested reviewers:** `thecodacus` + +</dd> +</dl> +</details> + +<!-- walkthrough_end --> +<details> +<summary><strong>Merge Risk:</strong> <strong>🟡 Moderate</strong> · up to <code>fee66</code></summary> +<dl> +<dd> + +<!-- final_review_risk_start --> +<!-- final_review_risk_label +**Merge Risk:** **🟡 Moderate** · up to `fee66` +--> +<!-- final_review_risk_coverage:{"sourceCommitId":"fee666ce7b7b846a6d941992d0755f330331c83c","coveredCommitId":"fee666ce7b7b846a6d941992d0755f330331c83c","kind":"reviewed"} --> + +Large DOCX imports can make the portal temporarily unresponsive. Bound or move import-time parsing before merging. +<!-- final_review_risk_end --> + +</dd> +</dl> +</details> +<!-- pre_merge_checks_walkthrough_start --> + +<details> +<summary><strong>Pre-merge checks: <sup><picture><img src="https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-counter-passed-14-v2.svg" width="14" height="14" align="middle" alt="Passed" title="Passed"></picture></sup> 4 · <sup><picture><img src="https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-counter-failed-14-v2.svg" width="14" height="14" align="middle" alt="Failed" title="Failed"></picture></sup> 1</strong></summary> +<dl> +<dd> + +<strong><sup><picture><img src="https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-counter-failed-14-v2.svg" width="14" height="14" align="middle" alt="Failed" title="Failed"></picture></sup> Failed checks: 1 warning</strong> + +| Check name | Status | Explanation | Resolution | +| :---------------- | :----------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------- | +| Docstring Coverage | ![Warning](https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-warning.svg) | Docstring coverage is 58.18% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 440 functions across 71 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. | + +<details> +<summary><strong><sup><picture><img src="https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-counter-passed-14-v2.svg" width="14" height="14" align="middle" alt="Passed" title="Passed"></picture></sup> Passed checks: 4 passed</strong></summary> +<dl> +<dd> + +| Check name | Status | Explanation | +| :------------------------ | :--------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------- | +| Description Check | ![Passed](https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-passed.svg) | Check skipped - CodeRabbit’s high-level summary is enabled. | +| Title check | ![Passed](https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-passed.svg) | The title clearly and concisely identifies the main change: adding an OnlyOffice Document Server extension for Office documents in the portal. | +| Linked Issues check | ![Passed](https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-passed.svg) | Check skipped because no linked issues were found for this pull request. | +| Out of Scope Changes check | ![Passed](https://storage.googleapis.com/coderabbit_public_assets/pre-merge-checks-passed.svg) | Check skipped because no linked issues were found for this pull request. | + +</dd> +</dl> +</details> + +</dd> +</dl> +</details> + +<!-- pre_merge_checks_walkthrough_end --> +<!-- finishing_touch_checkbox_start --> + +<details> +<summary><strong>✨ Finishing Touches: 💡 2</strong></summary> +<dl> +<dd> + +<!-- finishing_touch_suggestion:docstrings --> +<details open> +<summary><strong>📝 Generate docstrings: 💡</strong></summary> +<dl> +<dd> + +- [ ] <!-- {"checkboxId":"3e1879ae-f29b-4d0d-8e06-d12b7ba33d98"} --> Commit to this branch +- [ ] <!-- {"checkboxId":"7962f53c-55bc-4827-bfbf-6a18da830691"} --> Create a new PR + +</dd> +</dl> +</details> +<details open> +<summary><strong>🧪 Generate unit tests (beta)</strong></summary> +<dl> +<dd> + +- [ ] <!-- {"checkboxId": "6ba7b810-9dad-11d1-80b4-00c04fd430c8", "radioGroupId": "utg-output-choice-group-unknown_comment_id"} --> Commit to this branch +- [ ] <!-- {"checkboxId": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "radioGroupId": "utg-output-choice-group-unknown_comment_id"} --> Create a new PR + +</dd> +</dl> +</details> + + +<!-- finishing_touch_suggestion:fix_ci --> +<details open> +<summary><strong>🛠️ Fix failing CI checks: 💡</strong></summary> + +- [ ] <!-- {"checkboxId": "9f0d24fb-b419-4f01-baf0-8b26b6424f34", "radioGroupId": "fix-ci-output-choice-group-6373469052"} --> Commit to this branch +- [ ] <!-- {"checkboxId": "6d21cfe8-ec3f-40e2-9222-b8318b64d3b0", "radioGroupId": "fix-ci-output-choice-group-6373469052"} --> Create a new PR + +</details> +</dd> +</dl> +</details> + +<!-- finishing_touch_checkbox_end --> + +<!-- autopilot:start --> +- [ ] <!-- {"checkboxId":"2708ad07-9f24-4260-9c11-7dc76a49f2e3"} --> <strong title="Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts">Autofix</strong> · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts +<!-- autopilot:end --> +<!-- tips_start --> + + + +Thanks for using [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=thecodacus/pithagoras&utm_content=307)! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. + +<details> +<summary><strong>❤️ Share</strong></summary> +<dl> +<dd> + +- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) +- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) +- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) +- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code) + +</dd> +</dl> +</details> + + +<sub>Comment `@coderabbitai help` to get the list of available commands.</sub> + +<!-- tips_end --> + +--- +## commenter: Piggidragon | created: 2026-10-23T21:50:05Z +Addressed all 13 review comments in d6a62fdf (replied on each). + +On the earlier CI failure of this branch (run 40014812580): the server job failed one assertion — the portal's shipped-extensions list now includes `pithagoras-office`, which `core/server/test/extension-ui.test.mjs` did not expect. Fixed in the same commit: the package moved to `extensions/office`, following the folder convention of its siblings, and the test's expected list names it. + +One note for maintainers: locally on a machine with an ambient `ALLOW_OPEN=1` (and portal credentials) in the environment, `core/server/test/auth-streams.test.mjs` fails one case because the harness spreads the parent environment into every test server; with a clean environment everything passes. + +--- +## commenter: Piggidragon | created: 2026-10-23T22:35:32Z +Round-2 review — all five findings addressed in aae4892: + +- callback.ts: JWT verified exactly once per request; the result is reused for the signed-envelope binding (no more expiry window between two verifications). +- ooxml readZip: 1 GiB archive-level cap on the combined advertised expansion, enforced in the central-directory loop before any entry is retained — many just-under-the-256-MiB-limit entries can no longer stack up to multi-gigabyte decompression (per-entry cap kept as a second layer). +- ping first-save rule: provenance now decides. An imported document (DocMeta.imported, stamped by importFile) baselines its pre-existing comment history without wakes; a created blank document has no baseline, so a mention written in the first editing session wakes the owner on that first save. +- office_read: docx details bounded independently of the 200k-char text cap (100 comments x 500 chars + exact commentCount), since details land in the tool-event store/web transcript unbounded by MAX_READ_CHARS. +- **Out-of-diff finding (tools.ts, forcesave-first hook): now fail-closed.** The hook resolves true only when a force-save is actually confirmed. In v1 a browser-side save cannot be confirmed from the server, so makeForcesaveHook returns false while any session is open (recording an audit line that says exactly that) and office_edit refuses with error:session_open — it no longer proceeds on an unconfirmed state. Once the hook is unavailable or unconfirmed, nothing is written and the revision gate is re-checked only after a confirmed flush. + +Suite: 123 tests, 0 failures (3 docker-gated skips); tsc clean. diff --git a/tasks/pr307-coedit-model.md b/tasks/pr307-coedit-model.md new file mode 100644 index 000000000..b51499b4f --- /dev/null +++ b/tasks/pr307-coedit-model.md @@ -0,0 +1,136 @@ +# PR #307 — co-edit model (no switch) implemented + +Branch `office/280-pithagoras-office`. The refuse-while-a-session-is-open model (`coEdit` setting + forcesave-first +hook + `error:session_open`) is removed. The agent is now just another user in the editor: edits ALWAYS proceed under +the existing strict mtime/revision gate (zero tolerance, `store.persistBytes`), and a stale edit fails loudly as a +CONFLICT (`error:revision_mismatch`, same code other tools/tests depend on) with a re-read-and-retry instruction, +naming the open human editor when one stands. `live.ts` behavior is untouched; open editors still learn about external +changes through the existing polling loop (`pollState`). + +## 1) Session-open / forcesave gate removed from office_edit — `extensions/office/src/tools.ts` + +- Module header, revision-model paragraph (now ~lines 16–25): rewritten to the co-editor model (no switch, no session + check; gate decides; stale edit → loud CONFLICT → re-read + retry). +- Injection-seams paragraph: dropped the "setForcesaveHook installs…" sentence. +- Forcesave seam deleted (was ~lines 75–93): `let forcesaveHook`, `export function setForcesaveHook(...)`. + `__resetToolsForTests()` now only resets `cfg` (line ~76). Import of `loadSettings` removed (no other user in this file). +- `office_edit` tool description (~line 328): rewritten — "You are a CO-EDITOR alongside any human who has the document + open — there is no session check and no switch… if the document moved on since (for example a co-editor's save) NOTHING + is written and you hear a CONFLICT with the fresh revision — re-read and retry." +- The old `if (loadSettings(...).coEdit && usersOf(...).length > 0) { …session_open refusals… }` block (~370–381 in the + pre-change file) is GONE. In its place, a comment (lines 354–356) stating presence neither allows nor blocks; the gate + alone decides. +- Stale-message now an explicit CONFLICT (`stale()` closure, lines 357–362): code stays `revision_mismatch`; message is + `CONFLICT: "<title>" is at revision <fresh>, not <base>: it changed since you read it. Nothing was written.` + when + `usersOf(docId).length > 0` an appended `openEditorNote`: "A human editor (names) still has the document open…". +- New catch around `persistBytes` (lines 379–392): if the strict final gate throws `{code:"revision_mismatch"}` (file moved + between the pre-gate read and the write), office_edit re-throws a loud CONFLICT `Refusal("revision_mismatch", …)` with the + FRESH `currentMtimeMs`, base revision, open-editor note when `usersOf(docId).length > 0`, and the re-read-and-retry + instruction. All other errors propagate unchanged (so `failed()`'s generic coding still applies everywhere else). + +## 2) forcesave hook removed from the server — `extensions/office/src/server.ts` + +- Header item 3 (line 9): replaced with a note that there is NO co-editing switch and NO hook to wire (co-editor model). +- Boot: `setForcesaveHook(makeForcesaveHook())` call deleted; import now `import { registerTools } from "./tools"` (line 49); + a NOTE comment at the seams (lines ~517–518) documents that no flush-before-write hook exists. +- `makeForcesaveHook()` function + its audit line deleted (was ~lines 137–152); section header is now just "settleKept" + (~line 126). + +## 3) coEdit setting removed end-to-end + +- `src/settings.ts`: `OfficeSettings` = `{ enabled, imageTag }` (line 13); `DEFAULT_SETTINGS` without `coEdit` (line 16); + `loadSettings` returns only the two fields. LEGACY tolerance is structural: `field()` only reads known keys, so an + existing settings.json carrying `"coEdit": …` loads fine and the value is ignored; doc comment at line 25 says exactly this. + `saveSettings` already dropped unknown/absent keys, so a re-save rewrites legacy files clean (covered by tests). +- `src/server.ts`: GET /state payload no longer carries `coEdit` (`res.json({ enabled, imageTag, container, docs, locks })`, + ~line 290); POST /settings: the `if ("coEdit" in body)` branch + `invalid_coedit` are deleted — a body of only `{coEdit}` + falls through to the existing 400 path (error text updated to `send { enabled?, imageTag? }`, line 441); audit line at + line 444 no longer logs coEdit; route doc comment notes there is no co-editing switch (line 424). +- `src/web/api.ts`: `OfficeState.coEdit`, `SettingsPatch.coEdit` and `SettingsAnswer.settings.coEdit` removed. +- `src/web/settings.tsx`: the "Live co-editing between agent and editor" `SwitchRow` + tooltip deleted; header comment + (lines 8–10) updated to state there is no switch. Surrounding layout untouched. +- The remaining "co-editing **keys**" strings in settings.tsx/server.ts (`:latest` warnings, lines 51/52/118 and the + `invalid_image_tag` error text) are a DIFFERENT concept — the DS JWT key — and were intentionally kept (behavior + tests + depend on that wording). + +## 4) Docs rewritten to the co-editor model + +- `src/tools.ts` module header (see §1): "The agent is a CO-EDITOR alongside human editor sessions (live.ts) -- like Office + 365 there is NO switch and NO session check: the edit proceeds under that gate regardless of who else has the document + open… loud CONFLICT (error:revision_mismatch), naming the open editor session when one stands — re-read and retry." +- `src/server.ts` header + boot NOTE (see §2). +- `README.md` line 11 ("Co-edit protection, on by default (`coEdit`)") replaced with "**Co-editor model (no switch)**: … + Conflicts are surfaced, not prevented by refusal. Open editors learn about external changes through the existing 5 s + polling loop (`live.ts pollState`). Every agent revision is recorded for the editor's version history." +- README Deferred item "Co-editing refresh handshake / M0 browser-reach decision" (line 35) kept VERBATIM — live in-editor + refresh of agent changes is still deferred and the entry stays accurate. +- `src/live.ts` comments fixed where they described the removed refusal: line ~29 (TTL no longer "refuses every office_edit"; + it keeps DS lock + badge honest), lines ~31–34 ("does NOT change lock/badge semantics… Agent edits are gated purely on + the revision, never on who is present — an open editor neither allows nor blocks office_edit"), line ~85 (usersOf doc). + **No behavior change in live.ts** (presence, TTL, `pollState`, `setLocked` untouched). + +## 5) Tests changed/added + +- `test/tools.test.mts`: + - Import of `setForcesaveHook` removed; header comment + `bumpRevision` doc updated. `bumpRevision` now mirrors the stat + READBACK into the index (this guest fs reports utimes'd mtimes ~1 µs low — otherwise the zero-tolerance gate is flaky). + - The old "forcesave-first" test (4 hook-based cases, incl. `error:session_open`) replaced by + **"office_edit co-editor model: a live human session neither blocks nor gates; a stale revision is refused with a loud CONFLICT"** + (~lines 294–340): Case A **INVERTED** — matching revision + open `live.setUser` session → edit SUCCEEDS; Case B — after an + external save advanced the mtime, the same edit is REFUSED: `/^error:revision_mismatch /`, contains `CONFLICT`, contains + the fresh mtime, names the open editor ("human editor (Ada)"), nothing written; Case C — same stale revision with no + session → still refused, no human-editor note (the gate, not presence, is the safety mechanism); Case D — re-read (fresh + base) → retry succeeds. +- `test/server.test.mts`: + - `makeForcesaveHook` removed from imports; its unit test deleted (settleKept test in that section kept). + - GET /state: payload keys asserted as `["container","docs","enabled","imageTag","locks"]` — no coEdit. + - POST /settings test: seeds a LEGACY settings.json with `"coEdit": true` before any request (loads without crashing); + new assertion block — `POST /settings {coEdit:true}` → 400 "no known setting…", and the follow-up GET /state payload + contains no `coEdit`. +- `test/settings.test.mts`: expectations updated to the 2-field shape; NEW test "a LEGACY file carrying the removed + 'coEdit' key loads without crashing and simply ignores it" (load returns clean shape, and a save rewrites the file + without the key); round-trip test now patches `imageTag` + a bogus key instead of coEdit. +- `test/integration.test.mts` (docker-gated, still must load/type): `setForcesaveHook` import removed; header item 3 and + scenario rewritten — Scenario A: open session + matching revision → edit SUCCEEDS alongside it; Scenario B: an external + co-editor save (`store.persistBytes`, the same path the DS callback persist takes) lands a newer revision after the agent + read → refused `/^error:revision_mismatch /` with `CONFLICT`, human bytes stay on disk. Revisions pinned via utimes + + index mirror for determinism (same fs µs drift as above). +- `test/container.test.mts`: fixture settings files no longer write a `coEdit` key. + +## Verification (from /root/repos/pithagoras) + +- `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → **138 tests: 135 pass / 0 fail / + 3 skipped** (the same 3 docker-gated integration skips; OFFICE_IT_DOCKER unset). Ran repeatedly (6+ runs) after the fs + drift fix — stable. +- `npx tsc --noEmit -p extensions/office` → **No errors found** (strict). +- `node scripts/build-extensions.mjs` → exit 0; regenerated the gitignored `web/` + `server.js`. + +## Grep proof — no coEdit / forcesave references remain for the removed mechanism + +Scope: `extensions/office/{src,web (regenerated bundle),README.md,test}`. + +``` +$ grep -rn "coEdit|session_open|setForcesaveHook|makeForcesaveHook" src/web/ README.md test/ (excluding the 3 legacy-tolerance tests) +src/settings.ts:26 doc comment explaining LEGACY "coEdit" key tolerance (intentional; names the on-disk key) +(nothing else — zero hits in src/web/**, README.md, and all other src + tests) +``` + +Intentional remaining `"coEdit"` literals are ONLY the legacy-tolerance material itself: +- `src/settings.ts:25` (comment), `test/settings.test.mts:28/30/35/59-60`, `test/server.test.mts:486/524/529`. + +Regenerated `web/` bundle: **zero** matches for `coEdit|session_open|setForcesaveHook|makeForcesaveHook`. + +"forcesave" survives ONLY as the OnlyOffice DS wire protocol, which was never ours to remove (kept per "keep all of +live.ts / existing callback persist behavior"): +- `src/callback.ts:16/133/139/230(forcesavetype field)/361/363/383/413` — status-1 forcesave action + status-7 handling. +- `test/callback.test.mts:258/262/275/283/441/452` — tests of that same DS behavior. + +"co-editing keys" (JWT pinning warning) strings in `src/web/settings.tsx:51,52,118`, `src/server.ts:452` and +`src/office-ui.ts:19` are the DS JWT-key concept, not the removed switch — kept (behavior + existing tests depend on it). + +## Git state / hygiene + +- No commit/push/checkout/stash/reset. 12 tracked files modified under `extensions/office/` (list via `git status`): + README.md; src/{live,server,settings,tools}.ts; src/web/{api.ts,settings.tsx}; test/{container,integration,server,settings,tools}.test.mts. +- Untracked files (tasks/, PLAN-office-280.md, .office-contracts.md, .pi/, .reports/) untouched. No temp files created in + the tree (one /tmp probe + /tmp logs, deleted). `web/` and `server.js` are gitignored build output, regenerated by the build. diff --git a/tasks/pr307-cr-latest.md b/tasks/pr307-cr-latest.md new file mode 100644 index 000000000..3e1cc2a56 --- /dev/null +++ b/tasks/pr307-cr-latest.md @@ -0,0 +1,75 @@ +# PR #307 — latest CodeRabbit comment review: import-time DOCX parsing on the portal event loop + +**Scope:** CodeRabbit thread id 4364892451 (2026-00-10-23, `extensions/office/src/store.ts:384`), 🩺 Stability & Availability / 🟠 Major / "🏗️ Heavy lift", plus its embedded ast-grep CWE-22 note on the same line. Review-only; no fixes implemented. + +## The comment + +> Keep import-time DOCX parsing off the portal event loop. The portal router handles `/import` by calling `store.importFile(...)` synchronously. `importFile` reads the complete source with `fs.readFileSync` and parses the DOCX inline. A large valid DOCX can block other portal requests. Move parsing to a worker or enforce a suitable import-size limit before the synchronous read. + +Suggested prompt: move the read+parse to a worker **or** enforce an import-size limit before `fs.readFileSync`; preserve `importedBaseline` behavior. + +Nested ast-grep note: `384-384: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal [CWE-22]`. + +## Evidence (file:line) + +### 1. The code path — claim is factually correct + +- `extensions/office/src/server.ts:366` — `r.post("/import", json, ...)`: takes `{path}` from the JSON body, `path.resolve(given)` (server-side, absolute or relative to portal cwd), gates with `portal().folders.allowed(parent)` (`server.ts:372`), then calls `store.importFile(abs, ...)` **synchronously inside the async handler** (`server.ts:376`). There is **no size limit anywhere on this route**. +- `extensions/office/src/store.ts:362` — `importFile`: stats the source (`store.ts:371`, `fs.statSync(absPath)`), rejects non-files (`:376`), then for docx does `importedBaseline = parseDocx(fs.readFileSync(absPath))...` (`store.ts:384`) and finally `fs.copyFileSync(absPath, file)` (`store.ts:407`). The xlsx branch skips parsing (copy only). +- **No import-size cap exists anywhere.** There is no HTTP upload at all — the client (`extensions/office/web/api.ts:77`, `web/client.tsx`) sends only a *path* string; the bytes are read from server disk. So "upload size limit" infrastructure does not apply; the file can be any size present in an allowed workspace folder, and the store header (`store.ts:6-7`) explicitly says the portal's 1 MB managed-file cap **does not apply** to the library (>1 MB Office files work natively by design). +- Portal process model: `extensions/office/package.json` has no `"isolated"` flag (only `pithagoras.server` + `requires: pithagoras-docker`), and isolated halves "give nothing to build on" (`core/server/src/extension-host.ts:887`) — so the office server half runs **in the portal's own process**, whose event loop it shares with core and every non-isolated extension (routes mounted at `/api/x/pithagoras-office`, `server.ts:12`). A blocking import therefore does starve all other portal requests. Claim valid. + +### 2. Worst-case parse cost (is it really big?) + +- `extensions/office/src/ooxml.ts:90,96` — `MAX_ENTRY_BYTES = 256 MiB` per entry, `MAX_TOTAL_BYTES = 1 GiB` total *advertised* expansion, enforced in `readZip` **before** any inflate. So memory is bounded (≈1 GiB resident worst case — `readZipParts` retains every expanded part), but that bound is large and: + - `inflateRawSync` on a ~256 MiB `word/document.xml` is single-threaded sync work (seconds of CPU); + - `findParagraphs`/`paragraphText`/`parseComments` then run multiple regex passes over the full XML string; a hostile-but-valid document can keep those multi-second (our own ooxml review measured ~18 s CPU / +523 MB heap for a **1.7 KB** xlsx via the dense-grid path — see cross-reference). + - `store.ts:407` `copyFileSync` then synchronously copies the whole file again. +- Practical worst case today: an agent-generated (or user-placed) several-hundred-MB docx in a workspace folder → one `/import` can block the shared portal event loop for seconds-to-tens-of-seconds and hold ~GiB of heap, while every other route hangs. Unbounded file size = unbounded blocking time. Claim valid, with real (if bounded) severity. + +### 3. House style — is this a regression? + +No. Sync fs in request paths is the established pattern across this codebase: +- `store.ts` module header declares it by design: "everything that touches existing files is sync"; sync is load-bearing for index atomicity ("no await may sit between readIndex and writeIndex", `createDoc`; same for `persistBytes`, which also does a full sync parse path via ping). +- Same extension, same process, same kind of work: `GET /documents/:id` downloads do `fs.readFileSync(fileFor(dd, doc))` of the **entire library file** (`server.ts:340`), `/export` reads back an exported copy (`server.ts:394`), the loopback listener reads bytes sync (`server.ts:215`). +- Core: ~30 core files use `readFileSync` on request paths (e.g. `core/server/src/api/*.ts`, `projects.ts`, `skills/*`); other extensions too (`extensions/git/src/git.ts:360`, `extensions/docker/src/docker.ts:51`, `extensions/office/src/tools.ts:190` — the `office_read` tool reads whole library files sync and then parses inline, i.e. agents can already trigger the same parse cost with **no** size limit). +- Our own review already judged this class "Acceptable at current scale" (see cross-reference below). + +So: a real availability hazard, but not a PR-307 regression — it is the house-wide sync-fs pattern made more visible because import has *no* size cap and adds a parse step. + +### 4. The nested CWE-22 note (path traversal) + +- `absPath` **is** request-derived (`server.ts:369-370`), but it is not free-form: `path.resolve` + parent must pass `portal().folders.allowed(parent)` (`server.ts:372`; definition at `packages/extension-api/server.d.ts:412-415`: agent home or inside the workspace root, "by its real path"). Arbitrary traversal (e.g. `/etc/passwd.docx`) is rejected with 403 — the classic CWE-22 framing overstates it. +- The genuine gap (ours, already filed): the check runs on the resolved-but-**not realpath'd** path, and `statSync`/`copyFileSync` (`store.ts:371,407`) follow symlinks — a symlink planted inside an allowed workspace folder points the import outside it. Our `pr307-security.md` N1 rates impact **low** because every principal (agents, portal) already has host-filesystem access as the same user; isolated-half.ts states the isolation "is no security boundary: the process is the portal's user". No conflict with CR — we never called it safe; we rated it low-value. A `fs.realpathSync` + re-run `folders.allowed` (≈4 lines) closes it either way and makes a clean response to both comments at once. + +### 5. `importedBaseline` behavior & the ping fallback (the "preserve existing behavior" constraint) + +- Consumer: `extensions/office/src/ping.ts:235-240` — on first save of an imported doc, if `meta.importedBaseline !== undefined` it baselines exactly those ids; **if absent, it falls back to baselining everything this save holds** (comment at `ping.ts:238-240`: "When the field is absent (unparseable source) we fall back..."). So "skip parsing → leave `importedBaseline` undefined" already has defined, safe semantics (first save never wakes; slightly more conservative than an explicit baseline). + +## Verdict + +**Accept-and-fix — but with the minimal variant of CR's options, not a worker.** The core claim is valid: `/import` today has no size cap and the docx branch does an unbounded-length sync read + inflate + regex passes on the shared portal event loop (single in-process half; worst case seconds–tens-of-seconds + ~1 GiB heap, bounded only by `ooxml.ts` expansion caps). The CWE-22 note is our already-filed N1 (symlink follow past the logical-path `folders.allowed` check), low impact. + +Recommended minimal fix (~8 lines in `importFile`, using the `st` that already exists at `store.ts:371`): +1. **Parse gate (soft):** run `parseDocx(readFileSync(...))` only when `st.size <= 32 MiB`; above it, leave `importedBaseline` undefined — exactly the documented "unparseable source" fallback in `ping.ts`, so imported-baseline behavior is preserved and no legitimate large doc is rejected. 32 MiB compressed ≈ far beyond any real text-document; parse cost stays under ~100 ms. +2. **Absolute cap (hard):** reject with a new `code: "too_large"` above e.g. **512 MiB**, guarding the copy and the ooxml memory envelope regardless of extension (xlsx imports skip parsing but still `copyFileSync` the whole file sync). +3. Optional, cheap, closes CR's ast-grep note too: `fs.realpathSync(absPath)` before the `folders.allowed` check in `server.ts` (per our N1). + +**worker_threads is not justified**: it would break the deliberately-sync store contract (index atomicity), require a second parse pipeline for the equally-sync tool/persist paths (`tools.ts:190`, ping persist) to be consistent, and the margin case (skip-parse + cap) removes the blocking in practice. **CR's "🏗️ Heavy lift" label is overblown** — the minimal fix is a few lines; the worker option CR listed first is the heavy one and the wrong choice here. + +## Recommended response (ready to post) + +> Good catch on the missing size bound — agreed, and the fix is small rather than a worker: `importFile` already stats the source (`store.ts:371`), so I'll (a) run the docx baseline parse only up to a 32 MiB cap and leave `importedBaseline` undefined above it — which reuses the documented "unparseable source" fallback in `ping.ts`, preserving first-save baseline semantics — and (b) reject imports over 512 MiB outright, since the sync copy and the ooxml 1 GiB expansion envelope make anything larger a shared-event-loop hazard. For the traversal note: the path is gated by `folders.allowed` on its parent (`server.ts:372`), so arbitrary traversal is blocked; the real gap is symlink-following past the logical-path check — I'll `realpathSync` and re-check that closes it. + +## Cross-reference with our findings (no conflicts) + +- **`tasks/pr307-callback-concurrency.md`** (finding, ~line 64): "multi-MB docx parsed with the sync zlib codec **blocks the event loop for tens of ms per persist** — … Acceptable at current scale; if docs grow large, move diff/wake to a queue…" → we already accepted this hazard class on the *persist* path as scale-acceptable and prescribed the same escape valve CR proposes ("if docs get large, …move off"). Consistent: accept-and-fix-light is exactly our prior stance. +- **`tasks/pr307-ooxml.md`** (🔴 blocking, dense-grid `parseXlsx`): measured 1671-byte xlsx → +523 MB heap / ~18 s CPU; "there is no per-file parse guard … reachable from project-chat agents". Same unbounded-parse family, worse worst case than CR's docx framing, and it establishes the concrete parse-cost numbers used above. Also `ooxml.ts:90/96` caps (256 MiB entry / 1 GiB total) come from this review wave. +- **`tasks/pr307-security.md` N1** ("`importFile` follows symlinks — allowed-parent is checked on the logical path only", fix: `realpathSync` + re-check): CR's embedded CWE-22 note is this finding. No conflict — we rated it low-value (shared-user FS access), CR calls it traversal; both say "the check can be bypassed via symlink". Worth folding N1's one-line fix into the same response. +- `tasks/pr307-triage-architecture.md`, `tasks/pr307-tools-verification.md`: not relevant to this thread. + +## Open questions + +1. Is 512 MiB the right hard cap for *any* import, or should xlsx (no parse) be allowed larger than docx? (Copy is sync either way — argue for one shared cap.) +2. Should the same size gate eventually apply to `GET /documents/:id` (`server.ts:340`, whole-file `readFileSync`) and the `office_read` tool path (`tools.ts:190`) — same hazard, different routes; out of scope for replying to CR but a candidate follow-up ticket. +3. (Process note) pre-existing worktree `/tmp/pi-main-check [main]` present at `git worktree list`; not created by this review session, left untouched. No scratch files were created; no fixes applied. diff --git a/tasks/pr307-ds-proxy-fix2.md b/tasks/pr307-ds-proxy-fix2.md new file mode 100644 index 000000000..428e556d7 --- /dev/null +++ b/tasks/pr307-ds-proxy-fix2.md @@ -0,0 +1,55 @@ +# PR #307 — follow-up fix: DS proxy identity encoding + honest Content-Length + +Branch `office/280-pithagoras-office` (pushed tip before this change: `bfb5edc5`). Scope: `extensions/office/**` only. +No docker commands; no writes under `/var/lib/pithagoras`. + +## The bug (empirically proven against the live stack) + +The `GET /ds/<rest>` reverse proxy (`extensions/office/src/server.ts`, route formerly around lines 529–567) used +global `fetch` to forward to the DS origin and copied upstream headers — including `content-encoding` and +`content-length` — verbatim. + +Node's undici `fetch` **auto-decodes gzip bodies but leaves the upstream `content-encoding: gzip` header and the +COMPRESSED `content-length` in place**. Verified live on the running stack: + +``` +GET http://127.0.0.1/web-apps/apps/api/documents/api.js (default accept-encoding) + -> content-encoding: gzip, content-length: 13720, arrayBuffer(): 67458 bytes +(identity request): content-encoding null, content-length: 67458 +``` + +The handler copied both lying headers, then `res.end(<67458-byte buffer>)` → Node emitted +`Content-Length: 13720` for a 67458-byte payload. Browsers read the first 13.7 KB of api.js (invalid JS) and the +keep-alive stream was corrupted/reset — hence *"the editor script at /api/x/pithagoras-office/ds/web-apps/apps/ +api/documents/api.js could not be loaded"* in every remote browser. Reproduced end-to-end: with a valid portal +session cookie, `GET /api/x/pithagoras-office/state` → 200 while both `/ds/` URLs died mid-transfer. + +## The fix (extensions/office/src/server.ts) + +1. **Ask for an unencoded body** — the fetch in this route only now passes `headers: { "accept-encoding": "identity" }` + (upstream OnlyOffice nginx honours it, verified). Upstream then sends plain bytes with a truthful length. +2. **Never copy lying size/encoding headers** — removed `content-encoding` and `content-length` from the header-copy + list (kept: `content-type`, `cache-control`, `content-disposition`, `access-control-allow-origin`). The body is fully + buffered (`arrayBuffer` → `Buffer`), so Node computes Content-Length from the ACTUAL bytes. A route comment now + records WHY (undici decodes gzip but keeps compressed lengths; we ask identity and size the answer ourselves, so + header and payload are exactly consistent). +3. No other routes touched; refusal paths (`ds_not_configured` / `unresolvable_ds_origin` / `ds_unavailable`) unchanged. + +## Tests (extensions/office/test/server.test.mts) + +- The four existing proxy tests (pass-through of body/content-type/raw query, upstream 404, dead origin → 503 + `ds_unavailable`, unconfigured/container refusal) were reviewed: **none asserted content-encoding pass-through**, + so no assertion changes were needed. +- **New regression test** — `/ds proxy: an identity request answered with GZIP bytes + LYING headers still delivers + the FULL original payload`: a fake DS on ephemeral 127.0.0.1 that answers our `accept-encoding: identity` ask with + a gzip-compressed body plus lying `content-encoding: gzip` / `content-length: <compressed byte count>`. Asserts: + 200, upstream was actually asked `identity`, content-type passes through, NEITHER lying header is copied onto the + proxied answer, and the full original payload arrives byte-for-byte (undici decoded it — no truncation at the + lying length, no manual gunzip). + +## Finish line + +- `env -u NODE_ENV node --import tsx --test extensions/office/test/*.test.mts` ×3 → 0 fail (baseline was 150 total / + 147 pass / 3 docker-gated skips; this change adds one test) — see results below. +- `npx tsc --noEmit -p extensions/office` → exit 0. +- `node scripts/build-extensions.mjs` → exit 0. diff --git a/tasks/pr307-ds-proxy.md b/tasks/pr307-ds-proxy.md new file mode 100644 index 000000000..2e4a6eead --- /dev/null +++ b/tasks/pr307-ds-proxy.md @@ -0,0 +1,80 @@ +# PR #307 — DS traffic proxied through the portal (remote-browser fix) + +Branch `office/280-pithagoras-office`, base tip before change: `e5673b5d`. Scope: `extensions/office/**` only. +No docker commands were run against `pithagoras-office-ds`; no touch of `/var/lib/pithagoras` or the running portal (read-only recon only). + +## The problem (live-verified before the change) + +The user's browser is NOT on the server host, but `POST /editor-config` answered +`apiJs: http://127.0.0.1/web-apps/apps/api/documents/api.js` — an ABSOLUTE loopback URL built from the recorded +DS origin (`dsOrigin(dd)`), because the DS container runs with `NetworkMode=host` and nginx binds :80 on the host. +On-host that URL serves 200 (OnlyOffice nginx) — but a browser on another machine cannot dial it, so +`office-editor.tsx`'s `loadScript()` failed with *"the editor script at http://127.0.0.1/web-apps/... could not be loaded"*. + +## Why a portal proxy works (empirically verified) + +- **api.js is prefix-aware.** Its `getBasePath()` walks the document's `<script>` tags and returns + `script.src.match(/(.*)api\/documents\/api.js/i)[1]` — everything BEFORE `api/documents/api.js` in the loaded + script's src becomes the base of the editor iframe URL. So the SAME api.js loads identically from + `…/ds/web-apps/apps/api/documents/api.js`, and the browser then builds the iframe from that same prefix. +- **createIframe(config)** sets `iframe.src = getAppPath(config) + getAppParameters(config)`; for our config + (`type: "embedded"`) path = `getBasePath()`, and appMap word→documenteditor / cell→spreadsheeteditor gives + `<prefix>/web-apps/apps/<app>/main/index.html?dockey=…` — still under the proxy prefix. +- **The editor pages are fully relative.** The live DS page `/web-apps/apps/documenteditor/main/index.html` + references every asset relatively: `src '../../../vendor/requirejs/require.js'`, + `src '../../common/main/lib/util/fix-ie-compat.js'`, `href 'resources/img/favicon.ico'`, etc. Because all of them + resolve inside the same prefix, serving api.js + editor pages under `/api/x/pithagoras-office/ds/web-apps/…` + keeps every reference inside that prefix — no path rewriting needed, just faithful byte forwarding. +- **The SIGNED config is untouched.** `buildEditorConfig` (server.ts ~249–287) puts ONLY loopback URLs in it: + `document.url = http://127.0.0.1:<loopPort>/doc/<ticket>` and `editorConfig.callbackUrl = …/callback`. Those are + fetched by the DS server-side (host netns → loopback works), never by a browser, so they stay exactly as-is — + changing them would break signing semantics and DS behavior. `apiJs` is NOT part of the signed payload: the + client drops it (`office-editor.tsx` ~lines 76–77: `delete config.apiJs`) before calling DocEditor, and the JWT + is computed over the plain config only (asserted in `test/server.test.mts`, "editor-config (docx)" test). + +## What changed + +1. **Proxy route** — `src/server.ts` `registerRoutes`: new catch-all `r.get(/^\/ds\/(.*)$/, handler)` on the same + router core mounts at `/api/x/pithagoras-office` behind the session gate (`core/server/src/extension-host.ts:1043`, + `app.use("/api/x", extensionRoutes)` in `core/server/src/server.ts:1711`). Handler: + - recomputes the upstream path from the RAW `req.url` after the first `/ds/`, query string included verbatim + (express params may be decoded; an iframe's `?dockey=…` must not be); + - `origin = dsOrigin(dd)` — empty → 503 `{code:'ds_not_configured'}`; `container:` prefix → 422 in the same JSON + family as the existing `/editor-config` refusal (`{code:'unresolvable_ds_origin', origin}`); + - `fetch(origin + '/' + rest, { redirect: 'follow', signal: AbortSignal.timeout(15000) })`; network error → 503 + `{code:'ds_unavailable'}` + `logLine` (Node 23 global fetch); + - answers the upstream status; copies only `content-type`, `content-encoding`, `content-length`, `cache-control`, + `content-disposition`, `access-control-allow-origin` when present; body buffered via `upstream.arrayBuffer()` + and passed through UNTOUCHED (Node fetch does not decompress, so a gzip/br body plus its `content-encoding` + reach the browser as-is — no corruption); everything else dropped. GET-only, no state mutation. +2. **editor-config** — `apiJs` is now root-relative: `` `/api/x/${encodeURIComponent(portal().extension)}/ds/web-apps/apps/api/documents/api.js` `` + (`portal().extension` = package name `pithagoras-office`). The browser resolves it against the page origin — + exactly where the user reached the portal; `getBasePath()` captures that whole prefix. Everything else in the + answer (config fields, token) is unchanged. +3. **Docs** — server.ts header block: browser reaches DS via the `/ds` proxy while `document.url`/`callbackUrl` + stay loopback for DS-side fetches. README: "a portal proxy for DS traffic" removed from Deferred; the M0 + browser-reach bullet now records that it resolved to the same-origin proxy; one new Security-notes bullet on the + gate + single-destination forwarding. The 422 for `container:` origins stays documented. + +## Tests (new, in `test/server.test.mts`; harness extended minimally) + +`MiniRouter` now also accepts RegExp patterns (express regex routes match the whole raw url incl. query; groups → +`params[0..]`), and the fake portal in `boot()` carries `extension: "pithagoras-office"`. The existing apiJs +assertion was updated from the absolute loopback URL to the new root-relative proxy URL (the test still asserts +the signed config's document.url/callbackUrl are loopback). New fake-DS helper (`http.createServer` on an ephemeral +127.0.0.1 port, mirroring the `dsStandin` pattern in `test/integration.test.mts` ~line 175) with a seeded +`ds.json` pointing at it; asserts: +- **200 pass-through**: body byte-for-byte, `content-type`/`cache-control` copied, and the RAW query string + (`?dockey=a%2Bb~c&lang=en-US&x=1%20two`) arrived upstream unmodified (verified against the fake DS's recorded + `req.url` — no re-encoding); +- **upstream 404 passes through as 404** with the upstream body; +- **dead port → 503 `{code:'ds_unavailable'}`** + a log line left behind; +- **no ds.json → 503 `ds_not_configured`**, and **`container:` origin → 422 `unresolvable_ds_origin`** (same family as `/editor-config`). + +## Verification + +- `env -u NODE_ENV node --import tsx --test test/*.test.mts`, run three times: **150 tests / 147 pass / 0 fail / 3 + skipped** (the 3 docker-gated integration tests; baseline before this change was 146/143/3). +- `npx tsc --noEmit -p extensions/office` → exit 0. +- `node scripts/build-extensions.mjs` → exit 0 (refreshes the gitignored built outputs the running portal loads + from: `server.js`, `web/`). diff --git a/tasks/pr307-fixes-core.md b/tasks/pr307-fixes-core.md new file mode 100644 index 000000000..d390aa5e4 --- /dev/null +++ b/tasks/pr307-fixes-core.md @@ -0,0 +1,167 @@ +# PR #307 — CORE logic fixes (implemented) + +Branch `office/280-pithagoras-office`, scope `extensions/office/src/` + its tests. All items implemented in the +given order, suite green after each item. Inputs: `tasks/pr307-ooxml.md` (items 1, 3, 7) and +`tasks/pr307-security.md` S2/S3/N1/N2 (items 2, 5, 6). + +**Finish line (final state):** `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → +**135 pass / 3 skip (docker-gated) / 0 fail** (baseline was 120/3/0 — 15 new tests). +`npx tsc --noEmit -p extensions/office` → clean. `node scripts/build-extensions.mjs` → office extension builds. + +--- + +## 1) [BLOCKING] Dense-grid xlsx parse memory explosion + +**What changed:** `src/ooxml.ts` +- New caps (`src/ooxml.ts:456-457`): `MAX_SHEET_CELLS = 256 * 1024` (materialized product cap per sheet) and + `MAX_TOTAL_GRID_CELLS = 1024 * 1024` (combined grid budget across all sheets of one archive) — the cell-space + analogs of the existing `MAX_ENTRY_BYTES = 256 MiB` / `MAX_TOTAL_BYTES = 1 GiB` byte caps at lines 90/96. +- `parseXlsx` (`src/ooxml.ts:426`) creates one archive-wide `gridBudget` shared by every sheet (the unbounded-sheet-count + hole), passed into `parseSheet`. +- `parseSheet` (`src/ooxml.ts:478-503`): after collecting the kept cells it now shrinks the HEIGHT before any + allocation — `rows = min(maxRow, floor(MAX_SHEET_CELLS/maxCol), floor(budget.left/maxCol))`, decrements the budget, + and sets the pre-existing `truncated` flag when anything beyond the affordable rows exists. The dense fill loop now + iterates only `rows × maxCol`. Axis caps (`MAX_GRID_ROWS`/`MAX_GRID_COLS`) unchanged. + +**Why:** the grid allocated `maxRow × maxCol` of the farthest *kept* coordinates — a 1,671-byte xlsx with sparse cells at +row 20000 / col ~500 per sheet measured ~523 MB heap + ~18 s CPU (review finding 🔴); sheet count was unbounded. Now the +materialized work is bounded per sheet AND per archive (~a few MB of JS arrays max), and the skip is reported honestly in +`truncatedSheets`, which `office_read` already surfaces ("those far-out cells were not read"). + +**Tests:** `test/ooxml.test.mts` — "parseXlsx bounds the dense grid of sparse far-out cells": 3 sheets × 2 cells +each (A1 + cell at column 500, row 20000), built with jszip; asserts fast parse (< 2 s), bounded grid height per sheet +(< 20000 rows, A1 still readable), and `truncatedSheets` true for every sheet. All existing fixture round-trip tests unchanged/green. + +## 2) Import route guards + +**What changed:** +- (a) `src/server.ts:374-396` (`POST /import`): `fs.realpathSync(abs)` runs BEFORE the `folders.allowed` gate and the + gate checks the REALPATHED parent; the realpathed path is what gets imported. Missing file → `404 not_found` (same status + as before, now pre-check). Closes symlink traversal out of allowed folders (CWE-22 / S-audit N1). +- (b) `src/store.ts:387`: `importFile` hard-rejects sources above `MAX_IMPORT_BYTES = 512 MiB` (`src/store.ts:44`) with + `{code:"too_large"}`, before any copy; `CODE_STATUS` maps it to HTTP 413 (`src/server.ts:166`). Guards the sync + `copyFileSync` and keeps imports under the ooxml reader's 1 GiB zip envelope. +- (c) `src/store.ts:391-401`: the docx `importedBaseline` is computed only for sources ≤ `MAX_BASELINE_BYTES = 32 MiB` + (`src/store.ts:45`); above the cap it stays `undefined`, so `src/ping.ts:235-247` applies its documented "unparseable + source" first-save fallback (baseline everything the first save holds) — first-save wake semantics preserved. +- Doc comment on `importFile` updated (`src/store.ts:362-370`). + +**Why:** no size cap existed (N2), and the allowed-folder check ran on a logical path while stat/copy followed symlinks (N1). +All-SYNC store contract kept — no worker, no async added. + +**Tests:** `test/server.test.mts` — new "a symlink inside an allowed folder pointing OUTSIDE is refused": link in the allowed +workspace → 403 `not_allowed`, nothing imported; a link staying INSIDE still imports (201). `test/store.test.mts` — two new +tests: 600 MiB sparse file (`ftruncateSync`) → `too_large`, library empty; ≤-cap real docx stamps `importedBaseline` +(deepEqual `[]` for the template) vs a 33 MiB sparse `.docx` → `importedBaseline === undefined`. + +## 3) CRC-32 validation on zip read + +**What changed:** `src/ooxml.ts:116` (`readEntryData`): after decode + length check, +`if (crc32(out) !== e.crc) throw ooxmlError(\`CRC mismatch for ${e.name}: …\`, "invalid_zip")`. The CRC was read from the +central directory and never compared — stored-entry byte flips and deflate streams that still inflate to the advertised +length parsed silently, and `rewriteZip` re-persisted corruption with its original wrong CRC. Now every decode path +(import, office_read, office_edit persist) fails loudly per the extension's fail-loud posture. + +**Tests:** `test/ooxml.test.mts` — "a zip entry whose data no longer matches its CRC-32 is rejected loudly": builds the docx +with jszip in STORED mode (so a flipped payload byte breaks ONLY the CRC, not inflate), flips one byte of the stored +`word/document.xml` payload; asserts `parseDocx` throws `invalid_zip` with "CRC mismatch for word/document.xml" and that +`rewriteZip(corrupt, {})` fails too (corruption is never re-persisted). The clean STORED archive parses first, proving the +flip is what trips it. + +## 4) office_comments output cap + +**What changed:** `src/tools.ts` +- Hoisted `MAX_READ_CHARS = 200_000` to module scope (`src/tools.ts:187`) shared by `office_read` and `office_comments`. +- `office_comments` (`src/tools.ts:425-430`): the aggregated text is now capped exactly like office_read — + `slice(0, MAX_READ_CHARS) + "\n\n[output truncated by office_comments: it exceeds 200,000 characters]"`. The details + side-channel is bounded the same way office_read bounds its own (comments sliced to 100 × 500 chars, true total in + `commentCount`), since details also land in the tool-event store / web transcript. + +**Tests:** `test/tools.test.mts` — new test: a doc with 300 comments × ~750 chars (well over the cap); asserts the marker is +present, output length ≤ 200,000 + marker slack, `Comments (300)` still announced, `details.commentCount === 300`, +`details.comments.length === 100`, each detail text ≤ 500 chars. + +## 5) Presence lock TTL + +**What changed:** `src/live.ts` +- `DEFAULT_PRESENCE_TTL_MS = 15 * 60_000` (`live.ts:44`) — matches the 15-minute editor-config signature lifetime + (`CONFIG_TTL_MS`, server.ts:52). Configurable via `setPresenceTtl(ms)` / readable via `getPresenceTtl()` (ignores + non-finite/≤ 0 values); `resetLive()` restores the default. +- Internal `PresenceRecord = LiveUser & { expiresAt }` (the exported `LiveUser` shape is unchanged — no ripple to + server/tools/web). `setUser(joined=true)` keeps the original `at` (badge shows first open) but pushes + `expiresAt = now + ttl` on EVERY join/heartbeat (`live.ts:102-103`), so a live DS session (status-1 heartbeats) never + trips its own TTL, while an abandoned mint — api.js failed to load, tab closed before DS connected — expires. +- Expiry is enforced lazily on read: `usersOf` (`live.ts:82-90`) physically drops stale entries and returns only live ones; + `pollState` now counts presence through the same filtered path (`live.ts:135`). Stale locks therefore vanish from the + /state badge, `lockLine`, AND the `office_edit` refusal (`tools.ts:371` uses `usersOf`) without any timer or new call sites. + +**Why:** S2 — presence was minted at editor-config time for one shared seat and cleared only on terminal DS closes; a +failed/stuck open left a permanent lock that DoSes every agent edit of the doc. Co-edit BEHAVIOR is otherwise untouched: +a non-expired open session still refuses agent edits (test `server.test.mts:563-580` "refusal while a session is open" +passes unchanged — default TTL is 15 min). + +**Tests:** new `test/live.test.mts` (4 tests): expiration drops the lock after a short forced TTL; heartbeats keep a live +session present past its original mint (and the badge's `at` stays the first join); TTL setter rejects nonsense values and +the default is 15 min; explicit lock flag / `clearDoc` semantics unchanged. + +## 6) Token purpose separation (S3) + +**What changed:** claim-based separation — no key change, no container/impact on adoption: +- `src/jwt.ts`: `signJwt(..., purpose?)` (`jwt.ts:37,48`) embeds a SIGNED `"typ"` claim in the payload; `verifyJwt(token, secret, purpose?)` + (`jwt.ts:58+`) requires that claim to equal the requested purpose exactly when one is asked for. Tokens without a typ + claim (DS-minted callback JWTs, and tokens from before this change) verify as before when no purpose is requested — + the existing 15-minute verification flow and already-running DS containers are untouched. +- Mint sites tagged: editor-config → `"editor-config"` (`src/server.ts:435`); the 60 s outbound-fetch token → + `"fetch"` (`src/callback.ts:463`). The DS-verified callback path (`callback.ts:307`) verifies WITHOUT a purpose — DS + cannot be made to mint our claims; its envelope binding already rejects non-callback tokens. +- `src/tickets.ts`: tickets have no JWT claims, so the family marker lives inside the MAC'd message — HMAC now over + `"document-ticket|docId|expMs"` (`tickets.ts:13,29,51`), mint + verify together. Token payload layout is unchanged. + +**Why:** one shared secret currently covers all three token families; an in-portal verifier can no longer be handed a +token of the wrong family, and each family's keying is distinct. A full-secret leak still forges everything (HMAC key +shared by design with the DS container) — see "not done" below. + +**Tests:** `test/jwt-tickets.test.mts` — two new tests: purpose claims separate families both directions + a stripped/ +rewritten typ claim fails verification of its family; ticket round-trip intact while a MAC computed over the un-prefixed +(legacy) message is rejected. All existing jwt/ticket/editor-config/callback tests pass unchanged (backward compat). + +## 7) [trivial] rewriteZip case-insensitive part dedupe — DONE + +**What changed:** `src/ooxml.ts:520-551`: change keys are first deduped against each other case-insensitively +(last-one-wins, jszip behavior), then matched to existing entries by lowercased name; a match differing only in case +OVERWRITES the part IN PLACE (original entry name and order kept) instead of appending a shadow duplicate. Input zips +that already contain a shadow pair get the data on the first occurrence only. Doc comment documents the contract. + +**Tests:** `test/ooxml.test.mts` — new test: `"WORD/DOCUMENT.XML"` over an archive holding `word/document.xml` replaces in +place (entry name list identical to the original, reparse shows the edit) and two case-variant change keys dedupe to the last one. + +--- + +## Deliberately NOT done (with reason) + +1. **Per-purpose derived keys (HMAC(secret, purpose))** for item 6 — not used: the editor-config signature is verified by + the DS container itself against `JWT_SECRET` as given (`container.ts` env), so any key derivation breaks existing and + adopted containers; the brief prefers the claim approach for exactly this reason. Residual risk unchanged for a full + secret leak (one HMAC key underlies all families) — same trust statement as S3's "acceptable for a single-user local + tool"; a rotate-secret setting is out of this slice's scope (server/container territory). +2. **DS callback JWTs carry no purpose claim** — DS mints them in its own envelope shape; we cannot add claims there. + The verify side still distinguishes families for every portal-minted token, and the signed-envelope binding at + `callback.ts:309+` already rejects config/fetch tokens arriving at the callback route. +3. **Lazily minting presence from the first DS-confirmed status-1** (S2's fuller fix) — only the TTL was added, per the + brief ("add a TTL … do NOT change co-edit semantics beyond adding expiry"). The early-unlock-with-two-tabs note in S2 + (a terminal close wipes both tabs' presence under one shared seat id) is unchanged by design; it remains documented + there. +4. **Wall-clock guard for `office_read` on huge legitimate workbooks** (ooxml review open question 2) — the grid bounds + are size-based and deterministic (≤ ~1 M cells per archive ≈ a few ms–tens of ms of fill); a timing guard would add + nondeterminism without protecting against anything the caps don't already bound. +5. **Nested `w:p` / `mc:AlternateContent` fix and fixture-timestamp determinism** (🟡/🟢 ooxml findings) — outside this + brief's item list; not touched so as not to destabilize the codec in the same pass. + +## House-style compliance + +- Everything stays SYNC (no worker, no new async seams); no new runtime dependencies; strict TypeScript clean. +- Co-edit behavior unchanged (coEdit ON still refuses agent edits while a *live* session is open) — only expiry added. +- No git state was changed; `tasks/`, `PLAN-office-280.md`, `.office-contracts.md`, `.pi/`, `.reports/` untouched; no temp + files left (tests use per-test `mkdtemp` dirs under `os.tmpdir()` per existing convention). Pre-existing working-tree + modifications to `extensions/office/README.md`, `build.mjs`, and `src/web/settings.tsx` predate this work (earlier + review rounds) and were not modified here. diff --git a/tasks/pr307-fixes-docs.md b/tasks/pr307-fixes-docs.md new file mode 100644 index 000000000..72f9ecf8b --- /dev/null +++ b/tasks/pr307-fixes-docs.md @@ -0,0 +1,200 @@ +# PR #307 review fixes — DOCS, WIRING & small SRC hygiene (docs agent) + +Scope: items 1–6 of the review + item 7 (checked, skipped). Core-logic items (ooxml.ts +parsing/CRC, server.ts import route, live.ts TTL, token claims) were owned by another agent +working in the same tree concurrently; I did not touch those files' logic. + +Branch: `office/280-pithagoras-office`. All verification commands run from `/root/repos/pithagoras`. + +--- + +## 1) Co-editing docs inversion — DOCS ONLY (behavior untouched, confirmed intentional) + +Real v1 semantics (asserted by `test/server.test.mts:557-572`, implemented in +`src/tools.ts:377` gate + `src/server.ts:146` `makeForcesaveHook` which resolves **false** while +any session is open): `coEdit` ON (default) => agent edits **refused** with +`error:session_open` until the human editor session closes; `coEdit` OFF => edits proceed under +the bare mtime revision gate even with a session open. + +Rewritten spots (all were inverted — they claimed on=proceeds/forcesave, off=refused): + +- `extensions/office/README.md:11` — bullet now: "Co-edit protection, on by default (`coEdit`): + while a human has the document open in the editor, an agent edit is REFUSED with + `error:session_open` until the session closes … With `coEdit` switched off in settings, agent + edits proceed under the plain mtime revision gate even with an editor session open." +- `extensions/office/src/web/settings.tsx:109` — the toggle's detail text now says the same + (on = refused until session closes; off = proceeds under the revision gate). +- `extensions/office/src/tools.ts:16-24` (module header comment) — was "the agent does not + refuse -- it FORCESAVEs … An open session never blocks"; now describes v1 fail-closed refusal + and the coEdit-OFF exception. +- `extensions/office/src/tools.ts:328` (`office_edit` tool description, agent-facing) — was + "proceeds only once their unsaved changes are flushed by a COMPLETED force-save"; now states + the refusal-until-close semantics and the coEdit-off gate-only path. +- `extensions/office/src/tools.ts:367-372` (comment above the gate) — reworded to "fail-closed in + v1 … v1's real hook never confirms while sessions are open, so this branch in practice always + refuses until the editor closes; with coEdit off it is skipped and the gate decides." + +Deliberately NOT touched: `src/tools.ts` gate logic (~:377), `src/server.ts:146-152` +(`makeForcesaveHook`, whose docblock was already accurate), and the tests asserting the refusal. + +Tests: no behavior change; suite green after the item (comment/string-only edits). + +## 2) Dead deps.js wiring — mirrored the pithagoras-docs pattern + +`extensions/office/deps.js` was committed but imported by nothing; `ensureDependencies` never ran; +the built `server.js` kept a top-level external `import JSZip from "jszip"`, so a hand-copied +folder without `node_modules` died at module load with `ERR_MODULE_NOT_FOUND`. + +Changes: + +- **New `extensions/office/src/jszip.ts`** — lazy loader replacing the static imports. Exports + `jsZip(): Promise<JSZipStatic>` (one cached load per process; a failed load is NOT cached) and + `armJsZipInstall(promise)` which makes the first call await the boot-time install. The file uses + `import type JSZip from "jszip"` only — erased at build, so the bundle carries no static jszip + dependency. Runtime interop checked: Node's dynamic `import("jszip")` gives the package object as + `.default` (CJS main), constructable + `loadAsync` present. +- `extensions/office/src/store.ts:17` — `import JSZip from "jszip"` replaced by + `import { jsZip } from "./jszip"`; `templateDocx` (`store.ts:138`) and `templateXlsx` + (`store.ts:182`) now do `new (await jsZip())()`. All other store I/O stays sync (house rule). +- `extensions/office/src/tools.ts:40,389` — same for the one edit-path use: + `const zip = await (await jsZip()).loadAsync(buf);`. +- **New `extensions/office/deps.d.ts`** — type surface of the JS `deps.js` so strict tsc accepts + the import from TS source (esbuild bundles the real .js; tsc reads the declarations). +- `extensions/office/src/server.ts:48-49,531` — server entry now wires the boot install, exactly + as docs' `server.js:396-399` does for better-sqlite3: `void armJsZipInstall(ensureDependencies(logLine));` + right after `setPortal`. Boot does NOT wait (background install); only the first + template/import call blocks on it. Module-header docstring extended with step 8 documenting this. +- `extensions/office/build.mjs:17,20` — the inline package-lock diffing was replaced by + `import { missingDependencies } from "./deps.js"` + `missingDependencies(HERE, { dev: true })`, + mirroring `extensions/docs/build.mjs` (build-time and runtime checks now share one implementation). + +Tests added: **`extensions/office/test/jszip.test.mts`** (2 tests) — the armed install promise +gates the first load (still pending while unsettled, resolves to the real package after release); +the load is cached (a later non-settling arm cannot re-gate it); resolved shape has `loadAsync`, +constructs an archive, and exposes a version string. + +Verification performed: + +- `node scripts/build-extensions.mjs` succeeds; emitted `server.js` contains **zero** static + `import ... from "jszip"` / `require("jszip")` — only the deferred dynamic `import("jszip")`, + and the deps.js logic is bundled (`pithagoras-office/installs` symbol present, install log lines + present). Top-level externals are node builtins only. +- Decisive check: renamed `extensions/office/node_modules/jszip` away and imported the BUILT + `server.js` with plain Node → `IMPORT-OK, default export: function` (the pre-fix bundle fails + this import); restored afterwards. +- `deps.js` detection sanity: `missingDependencies()` → `[]` with jszip present, `["jszip"]` with + it hidden; i.e. a fresh boot would log-and-install in the background, and `jsZip()` would wait + for it. + +## 3) "Export to chat folders" claim removed + +The export route (`src/server.ts:390-407`) copies to a temp dir, serves an attachment and deletes +it; both library buttons (`src/web/library.tsx:155-160`) are plain downloads. No copy-into-chat- +folder feature exists. + +- `extensions/office/README.md:9` — now "import .docx/.xlsx into the library and export (download) + a copy of one". +- `extensions/office/README.md:38` — Deferred list gains "**Exporting into chat folders** — the + export route serves a download of the stored bytes (fresh copy, served as an attachment); copying + a document into a chat folder does not exist in v1." + +PR body untouched; feature NOT implemented. No tests needed (docs only). + +## 4) Second index writer eliminated + +`src/ping.ts` `addGrant` (old :152-173) did its own read-modify-write + rename of +`index.json`, bypassing the store's atomic path and duplicating the dedupe rule. + +- **New `extensions/office/src/store.ts:204-221`** — exported + `addGrant(docId, chatId, commentId, at, root?): "added" | "duplicate" | null`. It is THE index + writer for grants: fresh `readIndex`, dedupe by (chatId, commentId), atomic tmp+rename through + the store's private `writeIndex` (`store.ts:61-66`). Sync and log-free per the store module + contract. Returns tri-state so callers can distinguish "added" / "already granted" / "doc not in + index"; a corrupt index throws the usual `{code:"index_corrupt"}` like every other store op. +- `extensions/office/src/ping.ts:33,149-157` — the local writer is gone; ping's wrapper now calls + `storeAddGrant(...)` and keeps its original log line for the vanished-doc case (`office ping: + document <id> vanished from index between resolve and grant -- skipped`). Call site unchanged + (`processComment`, ~:216); a throw (corrupt index) is caught by the existing per-comment + try/catch in `handlePersisted` (`ping.ts:~265`) which logs and leaves the comment unseen for + retry — strictly safer than the old raw-parse, which crashed out of that path on malformed JSON. + +Behavior kept identical for all realistic inputs (well-formed index): same dedupe key, same grant +shape `{chatId, commentId, at}`, same file format (`JSON.stringify(index, null, 2)`, no trailing +newline — matches the store writer), and `writeSeen` still runs only after processing. Existing +ping/store tests (grant count, dedupe, `.at` typed, doc-missing skips) all pass unchanged; no new +test was added because the existing suite already exercises the grant path end-to-end through the +new single writer. + +## 5) Mention-grant wording softened to match reality + +Grants are permanent in v1 (per-comment expiry is deferred), so "for this task" was a false promise. + +- `extensions/office/src/ping.ts:104` — wake text last line: was + `"You have edit access to this document for this task; answer with the office_* tools."`, now + `"You have been given edit access to this document; answer with the office_* tools."`. The + `officeWakeMessage` docblock (`ping.ts:99`) now explains why ("given access", NOT "for this + task": grants are permanent, expiry deferred). +- `extensions/office/test/ping.test.mts:101,105` — the two exact-template assertions updated to + the new sentence. No expiry/revocation implemented (explicitly out of scope). + +## 6) Host-mode DS network exposure documented (no code change — untestable here) + +Container setup untouched (per brief; no OnlyOffice image available locally to verify nginx). +Added a **`## Security notes`** section to `extensions/office/README.md:40-42`: the DS container +runs with the portal's shared network (`NetworkMode` from `docker.sharedNetwork()`, resolved to +`"host"` in this deployment per `src/container.ts:194,211`), so the image's nginx serves the DS +web origin (:80) on 0.0.0.0 — host/LAN/tailnet peers reaching port 80 get the Document Server UI +with no portal login gate in front (the per-config JWTs sign portal-minted requests, they are not +access control for strangers). Operators should firewall :80 or (planned work) override +`/etc/nginx/conf.d` via a named volume. This also becomes the anchor for the pre-existing "see +below" pointer at README:28, which previously pointed at nothing. + +## 7) [Optional] Cross-paragraph quote edits leave a stray line break — SKIPPED (not a one-liner) + +Checked `src/ooxml.ts` `applyEdit` (:314-385): the replacement lands in the first touched `w:t`; +fully-covered runs in LATER paragraphs are dropped, but their now-empty `<w:p>` elements stay in +document.xml. Re-parsing then yields an empty paragraph (a stray `\n`) at that position. A correct +fix means removing fully-emptied `<w:p>` subtrees from the XML (while not touching paragraphs that +keep text, bookmarks or section properties) — real paragraph-level surgery with edge cases +(sectPr-bearing final paragraph), NOT a safe one-liner. Left for the core-logic owner; noted here. + +--- + +## Finish-line verification (final state of my items) + +- Tests: `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → + **0 fail, 3 skipped** (the docker-gated skips from baseline). Pass count grew 120 → 135: +2 mine + (jszip loader), the rest added by the concurrent core-logic agent in the shared tree. +- Types: `npx tsc --noEmit -p extensions/office` → **clean** (strict). Note: during my run window + the other agent had `src/live.ts` mid-edit (a transient `expiresAt` type error); at final check + it — and everything else — compiled clean. +- Build: `node scripts/build-extensions.mjs` → DONE; deps wiring verified as in item 2 (no static + jszip import in built server.js; imports without node_modules/jszip present). + +## Deliberately NOT done + +- Item 7 (see above): skip + note, per the "only if quick" rule. +- No bundle-of-jszip alternative for item 2: `packages/extension-api/build.mjs` hardcodes + `packages: "external"` for server builds, so inlining jszip would have meant bypassing the shared + builder — the docs pattern (lazy import + boot install) is the repo's established shape and was + mirrored instead. `deps.js` therefore KEPT (now wired), not deleted. +- Co-edit behavior, container/nginx config, grant expiry/revocation, export-to-chat-folders: all + untouched per brief; only docs changed where required. +- No temp files/dirs created; no git state-changing commands run. Untracked files in the tree + (`tasks/`, `PLAN-office-280.md`, `.office-contracts.md`, `.pi/`, `.reports/`) left as found. + +## Files I touched + +| File | Change | +| --- | --- | +| `extensions/office/README.md` | items 1, 3, 6 (bullets :9/:11, deferred :38, security notes :40) | +| `extensions/office/src/web/settings.tsx` | item 1 (:109 tooltip) | +| `extensions/office/src/tools.ts` | item 1 (header :16-24, description :328, comment :367) + item 2 jszip lazy import (:40,:389) | +| `extensions/office/src/server.ts` | item 2 boot wiring (:48-49,:531, header step 8) — no other server logic touched | +| `extensions/office/src/store.ts` | item 2 lazy jszip (:17,:138,:182) + item 4 new exported `addGrant` (:204-221) | +| `extensions/office/src/ping.ts` | item 4 single writer (:33,:149-157) + item 5 wake text (:99,:104) | +| `extensions/office/src/jszip.ts` | NEW (item 2) | +| `extensions/office/deps.d.ts` | NEW (item 2, types for deps.js) | +| `extensions/office/build.mjs` | item 2 (uses deps.js `missingDependencies`) | +| `extensions/office/test/jszip.test.mts` | NEW (item 2 tests) | +| `extensions/office/test/ping.test.mts` | item 5 (two template assertions) | diff --git a/tasks/pr307-ooxml.md b/tasks/pr307-ooxml.md new file mode 100644 index 000000000..f99407ac2 --- /dev/null +++ b/tasks/pr307-ooxml.md @@ -0,0 +1,83 @@ +# PR #307 review — slice: OOXML + sync zip codec correctness + +PR: "Office documents in the portal: OnlyOffice Document Server extension" (branch `office/280-pithagoras-office` @ fee666ce, base main). +Slice scope: `extensions/office/src/ooxml.ts`, `scripts/make-fixtures.mjs`, `fixtures/sample.{docx,xlsx}`, `test/ooxml.test.mts`, `test/fixtures.test.mts`. + +## Scope + +1. Sync zip reader/writer over `node:zlib`: stored vs deflated entries, CRC32, LFH + central directory + EOCD scan, endianness, name edge cases (UTF-8 flag bit 11, leading slash, case), zip64 handling, truncation/corruption, bad-CRC behavior, zip-bomb bounds. +2. OOXML primitives: docx/xlsx parsing, quote-based `editDocx`, sharedStrings vs inline strings, XML escaping, entity decoding, namespace/structure robustness. +3. Robustness of untrusted input and memory bounds; downstream trust assumptions (parts served to OnlyOffice DS / browser). +4. Reuse audit: existing zip/OOXML utilities in the repo; validity of the "jszip is async-only" claim. +5. Fixture determinism of `make-fixtures.mjs`. + +## Method + +- Read `src/ooxml.ts` (512 lines) in full, plus `scripts/make-fixtures.mjs`, `test/ooxml.test.mts`, `test/fixtures.test.mts`; grepped `core/` and `extensions/` for existing `zlib`/zip utilities; checked `package.json`/`deps.js` for the jszip claim. +- Empirical scratch (under `/tmp`, deleted before finishing): round-tripped both committed fixtures through the codec's `rewriteZip` and validated with python `zipfile.testzip()` (strict CRC + structure check); built zips with python (`ZIP_STORED` entries, leading-slash names) and read them with the codec; corrupted a stored-entry byte (bad CRC) and a deflate stream byte; hand-crafted sentinel zip64 EOCDs, truncated buffers, and a hostile central directory advertising a 300 MiB entry; built non-ASCII/emoji entry names; ran crafted docx bodies with nested `w:p` in `wps:txbxContent` and body-level `mc:AlternateContent`; measured heap/CPU of `parseXlsx` on a ~1.7 KB xlsx whose cells reference row 20000 × column 500 per sheet (3 sheets). +- Ran the repo's own suite for this slice: `node --import tsx --test extensions/office/test/ooxml.test.mts extensions/office/test/fixtures.test.mts` → **17/17 pass**. + +## Findings + +### 🔴 [blocking] Dense-grid allocation in `parseXlsx` scales with cell *coordinates*, not file size — a ~1.7 KB upload can allocate > 500 MB and burn ~18 s of CPU (multiple GB possible) + +- **Where:** `extensions/office/src/ooxml.ts:437-438` (`MAX_GRID_ROWS = 20000`, `MAX_GRID_COLS = 500`) and the dense fill at `src/ooxml.ts:461-470`; per-sheet loop `src/ooxml.ts:419-432` (no bound on sheet count); caller `extensions/office/src/tools.ts:296` (`office_read`, user/agent-triggered). +- **Why it matters:** The caps bound each *axis* per sheet, but the materialized grid is the **product** `maxRow × maxCol` of the farthest *kept* coordinates. A worksheet containing just two cells — one at column 500 and one at row 20000 — forces a 20000×500 = 10 M-cell array (~80-170 MB) per sheet, and there is no cap on the number of sheets. Measured: a **1671-byte** xlsx (3 tiny sheets) consumed **+523 MB heap and ~18 s CPU** in `parseXlsx`. 50 such sheets fit comfortably under any import size limit (`store.ts:6-7` explicitly states >1 MB Office files are allowed and there is no per-file parse guard) and would push several GB — an OOM of the shared portal process from one legitimate `office_read` call (and it is reachable from project-chat agents, i.e. by more principals than just the file owner). +- **Fix:** Bound the *materialized* work, not just the axes: pass a total-cell budget across all sheets (e.g. 100k cells) into `parseSheet`, stop densifying past it and report the sheet as truncated in `truncatedSheets` (the plumbing for this flag already exists). Alternatively keep rows sparse (`Map<number,string[]>`) and materialize only up to the `MAX_READ_CHARS` output budget at `tools.ts:297`. A per-sheet product cap (`maxRow * maxCol > N → truncation, early out before allocating`) is the minimal change. + +### 🟡 [important] No CRC-32 validation on read — corrupt archives parse silently and get re-persisted with their original (wrong) CRC + +- **Where:** `extensions/office/src/ooxml.ts:108-111` (`readEntryData`): stored data is returned as-is, deflated data only gets a *length* check after inflate; `e.crc` is read from the central directory (`src/ooxml.ts:67`) and never compared. +- **Why it matters:** Verified empirically: flipping one byte in a stored entry's payload makes `parseDocx` return success with garbage content, and a corrupted deflate stream can still inflate to exactly the advertised length (both produced no error). This contradicts the module's fail-loud posture (`bad deflate data`, `size mismatch`, truncated payloads all throw `invalid_zip`) — the one corruption class that is most common in the wild (bit rot, partial writes) slips through. Worse, `rewriteZip` then re-embeds the corrupt bytes with their original CRC, so the portal will keep "editing" a file that Word/DS report as corrupted, masking the root cause until it is much harder to diagnose. +- **Fix:** one line per entry in `readEntryData`: `if (crc32(out) !== e.crc) throw ooxmlError(\`CRC mismatch for ${e.name}\`, "invalid_zip");` — cheap (`out` is already fully materialized), makes import/parse reject damaged files loudly. + +### 🟡 [important] Nested `w:p` (text boxes, `mc:AlternateContent`) corrupts extraction and mis-targets quote edits; the doc comment describes different behavior than what happens + +- **Where:** `extensions/office/src/ooxml.ts:241-250` (`findParagraphs`, non-greedy `/<w:p(?:\s[^>]*?)?(?:\/>|>([\s\S]*?)<\/w:p>)/g`) with the same shape used by `applyEdit` at `src/ooxml.ts:316+`. +- **Why it matters (verified empirically on a crafted docx):** an outer `<w:p>` containing a drawing whose `wps:txbxContent` holds its own `<w:p>` truncates the outer paragraph at the *first inner* `</w:p>`. Consequences observed: + - runs after the textbox close in the same paragraph **vanish from extracted text** (`"AFTER"` absent; paragraph came out as `"BEFOREIN-BOX"` — box text glued to host text with no separator); + - `editDocx` then answers `quote_not_found` for that visible-but-dropped text, while quoting the *box* text succeeds and splices the replacement into the host paragraph's virtual text — an agent edit can silently intermix body and shape text; + - body-level `mc:AlternateContent` (Word's standard fallback duplication) emits **both** Choice and Fallback paragraphs, so duplicated legacy text appears twice in `parseDocx` output and will trigger spurious `quote_ambiguous` on edits. + Text boxes and AlternateContent fallbacks are routine in real Word files (stamps, shapes, ink/SmartArt). The doc comment at `src/ooxml.ts:241-242` ("such content would be read as the outer paragraph's trailing run") does not describe actual behavior — trailing runs are dropped, and box text is *included*. +- **Fix:** make `findParagraphs` nesting-aware: when a match encounters `<w:drawing>`/`wps:txbxContent` (or an inner `<w:p`) inside the paragraph content, either skip to the corresponding close tag before continuing the outer paragraph's scan, or detect nested paragraphs and throw a specific code (e.g. `nested_paragraph_unsupported`) so callers fail loudly instead of returning wrong text. At minimum, correct the doc comment and add a regression test; also decide explicitly how `mc:AlternateContent` Fallback content should be treated (skip `<mc:Fallback>` regions when `<mc:Choice>` is present). + +### 🟢 [nit] "Deterministic" fixture output is not — committed fixtures embed wall-clock timestamps, so regenerating churns the binaries + +- **Where:** `extensions/office/src/ooxml.ts:138` (`writeZip` stamps `dosDateTime(new Date())` when an entry has no explicit `time`); `scripts/make-fixtures.mjs:9` and `test/fixtures.test.mts:10` both claim "deterministic output"; committed `fixtures/sample.docx`/`sample.xlsx` carry the generation time `(2026, 10, 23, 19, 49, 8)` on every entry. +- **Why it matters:** re-running the documented regeneration command produces different bytes → git churn / accidental conflicts; the determinism claim (load-bearing for "fixtures double as round-trip material") is false as written. +- **Fix:** pass a fixed `time` in `make-fixtures.mjs` — `writeZip` already accepts explicit timestamps, so pin e.g. DOS epoch (`{ time: 0, date: 0x0021 }`) per entry and regenerate once. + +### 🟢 [nit] `rewriteZip` change keys that miss existing names only by case silently create shadow duplicate parts + +- **Where:** `extensions/office/src/ooxml.ts:196-208` (`rewriteZip` adds any unapplied `changes` key as a new entry; matching is exact-string, zip names are case-sensitive here, consistent with jszip). +- **Why it matters (verified):** rewriting `{ "WORD/DOCUMENT.XML": ... }` over an archive containing `word/document.xml` yields a zip with *both* parts; Word/DS treat OOXML part names with filesystem-ish case tolerance in several code paths, so a shadowed part can win or confuse the engine. Current callers build keys from exact parsed names so risk is low (speculative impact). +- **Fix:** cheap guard — reject `changes` keys that differ from an existing entry name only in case (`invalid_zip`/specific error), or log/warn; and keep a test documenting the exact-match contract. + +### 💡 [suggestion] Reuse audit: the sync codec is justified, but two zip implementations now have to agree by hand + +- Verified the claim: jszip (v3.10.1, `extensions/office/package.json`) exposes only Promise-based `loadAsync`/`generateAsync` — no synchronous API exists, so a sync reader/writer for the inline ping/tool paths is necessary. No existing zip utility in the repo (`core/server/src/compressed-file.ts` is gzip/brotli of web assets, unrelated). The codec therefore does not reinvent anything available; keep it. +- Suggestion: `store.ts:17` and `tools.ts:40` use jszip (templates) while runtime read/rewrite goes through this codec, and behaviors are already being deliberately aligned (`src/ooxml.ts:194`: "duplicate names: last one wins (jszip behavior)"). The cross-validation tests (ooxml.test.mts vs jszip, fixtures.test.mts) are good; consider extending them into a small property suite run in CI (random entry names/sizes through both writers, read by both + python) so future divergence on either side is caught. + +### 📚 [learning] Zip-bomb and format-robustness checks all hold up empirically + +- Bounds are size-based, not ratio-based, and checked against **advertised** sizes before any inflate: per-entry 256 MiB (`src/ooxml.ts:90`, enforced at `103`) plus a 1 GiB archive total summed in the CD loop *before* payloads are read (`src/ooxml.ts:66-69`). A hostile central directory advertising a 300 MiB deflated entry was rejected in **0.14 ms with ~zero allocation**. +- Endianness/CRC of the writer: python `zipfile.testzip()` passes on both fixtures and on stored/deflate round-trips (CRCs exact, little-endian fields correct); UTF-8 flag bit 11 is set exactly for non-ASCII names (verified with emoji + Scandinavian entry names). +- Data-descriptor archives work (CD sizes are authoritative, `src/ooxml.ts:98` comment), zip64 sentinels rejected (`src/ooxml.ts:53`, `70`), truncation/corruption/non-zip all throw `invalid_zip`, EOCD scan window is spec-correct (22+65535). +- Trust assumption: untouched parts are preserved byte-for-byte and served to DS/browser as-is — identical trust posture to opening any uploaded docx in Word; the extension adds no new attack surface of its own, and the regex-based parser has *no* entity-expansion or DTD attack surface (structurally safer than a full XML parser). Zip names are only map keys — no path traversal. Untrusted content is worth a line in the README rather than code: imported malicious OOXML parts behave exactly as they would in a local word processor. + +## What's good + +- **Empirical interop is clean:** both committed fixtures round-trip through `rewriteZip` and validate under python `zipfile.testzip()`; python-built zips (stored entries, leading-slash names) read back correctly; stored+deflate content survives byte-identical. +- **`editDocx` semantics are carefully engineered:** unique-quote contract with `quote_not_found`/`quote_ambiguous`, sequential ops seeing prior results, cross-run spans handled exactly (verified `"lpha be"` → `"Amiddleta"`), quotes covering only paragraph breaks rejected (`src/ooxml.ts:352-353`), `xml:space="preserve"` added precisely for leading/trailing spaces, and replacements escaped via `escapeXmlText` so `&`/`<`/`>` (and even entity-looking text) round-trip literally. A replacement containing XML special characters cannot inject tags — verified with the `"Delta & nine"` test path. +- **Dense-grid truncation reporting already exists** (`truncatedSheets`) — the 🔴 fix slots into existing plumbing rather than a new error surface. +- **Tests cross-validate across three implementations** (own codec, jszip, python via fixtures) and include a hand-crafted hostile central directory for the archive cap; 17/17 pass under `node --import tsx --test`. +- The honest "accepted deviation" documentation around non-byte-identical recompression (module header + `rewriteZip` doc) is exactly right. + +## Open questions + +1. Should import (`store.importFile`) hard-reject CRC-corrupt archives once the 🟡 CRC fix lands — or accept-and-warn? Currently xlsx imports never run `parseXlsx`, so corruption is only discovered lazily at first read/edit. +2. Even after bounding the grid (🔴), is a *wall-clock* guard warranted for `office_read` on very large but legitimate workbooks (16k rows × wide)? The 20000×500 axis caps truncate far-out cells per sheet (they already surface a truncation note to the agent via `tools.ts:303-304`) — acceptable for v1, but values beyond the cap are absent from the grid while still visible in the real file; is that enough honesty for quote-based edits against such sheets? +3. For M4 (xlsx comments via `xl/commentsN.xml` + VML): will the same regex approach be reused — if so, the nested-paragraph fix pattern from the 🟡 finding may matter there too (VML anchoring overlaps cell ranges). + +--- +*Review environment: node v22.23.2; all scratch confined to /tmp and removed after review.* diff --git a/tasks/pr307-panel-workspace.md b/tasks/pr307-panel-workspace.md new file mode 100644 index 000000000..f5122ee63 --- /dev/null +++ b/tasks/pr307-panel-workspace.md @@ -0,0 +1,71 @@ +# PR #307 follow-up — Office panel: shared workspace docs, compact create, sort by workspace + +Branch `office/280-pithagoras-office`, base tip before this work: f6788220. All changes confined to `extensions/office/**` (+ this untracked findings file). + +## What changed + +### 1. Unowned docs are shared (panel + every home's tool scope) +- `src/store.ts` — `docsForHome(homeChatId)`: condition is now + `doc.owner === null || doc.owner === homeChatId || doc.grants.some((g) => g.chatId === homeChatId)`. + JSDoc rewritten (unowned = workspace-scanned / human docs, visible to all homes; owned/granted unchanged). + (Also moved the JSDoc that had become orphaned above `addGrant` back onto `docsForHome`, where it belongs.) +- `src/client.tsx` — `OfficePanel` filter uses the same condition, so `/root/repos/pithagoras/.../fixtures/sample.docx` + (already indexed at boot with `owner: null`) now shows in every chat's sidebar panel. Empty-state text no longer + contradicts ("the workspace .docx and .xlsx files, human-made docs, and the ones this chat owns or was granted…"). +- `src/tools.ts` — module comment updated: the tool scope is `docsForHome(home)`, which includes every UNOWNED doc; + behavior follows from the store change, no per-tool code change needed. + +### 2. Compact create affordance in the panel +- `src/client.tsx` — new `PanelNewDocument()` at the top of the panel: title input + word/sheet icon kind toggle + (LuFileText/LuTable with `aria-pressed`) + Add button → same `api.createDocument({title, type})` as the page's + `NewDocument`, no owner passed (unowned semantics kept), then `refreshOfficeState()` so the doc appears + immediately; errors via `portal.ui.ErrorBanner` like everywhere else. +- `style.css` — slim sidebar chrome: `.po-newcompact` (one dashed row, wrapping) + `.po-kind` / `.po-kind-btn` / + `.po-kind-on` icon toggle using the portal theme vars. + +### 3. Sort by workspace +- `src/store.ts` — `DocMeta.workspace?: string`. `scanWorkspace` stamps it: FIRST path segment under whichever scan + root contains the file, **deepest matching root wins** (nested roots label more specifically than their parents); + a file straight in a root takes that root's OWN dir name (home-root file → home dir name; its own file name would + be noise). Stamped at index time via `indexOne(real, rootDir, workspace)` and BACKFILLED on every rescan for + pre-existing index entries (a backfill counts as a refresh exactly once, then idempotent). Library-created docs + (`createDoc`) stay undefined — legal, UI shows "library". `/state` already hands back DocMeta verbatim, so no + server route change. +- `src/web/api.ts` — `OfficeDoc.workspace?: string`. +- **new** `src/web/sort.tsx` — the sort machinery moved OUT of library.tsx and shared by page + panel: + `SortKey = "name" | "modified" | "size" | "type" | "workspace"` (+ `SortChoice`), `loadSort`/`saveSort` under the + SAME `"po-sort"` localStorage key (pattern unchanged; one added valid value), `workspaceLabel(doc)` → + `doc.workspace ?? t("library")`, `compareDocs(a, b, key, dir)` with `dir` applied INSIDE so "workspace" keeps its + undefineds LAST regardless of direction, and the `SortBar` component (now with a Workspace option) used by both. +- `src/web/library.tsx` — local sort defs/SortBar deleted in favor of the shared module; DocRow meta line now shows + `title.ext · when · workspaceLabel(doc)` so the label ("library" for undefined) is visible on the full page. +- `src/client.tsx` — `OfficePanel` holds `[sort, setSort] = useState(loadSort)`, persists via `saveSort`, and sorts + the filtered list with `compareDocs(a, b, sort.key, sort.dir)`; renders the shared `<SortBar>` above the rows. + Panel and full page therefore share one persisted choice. + +### 4. Copy audit +Grepped `src/web/*.tsx src/client.tsx` for user-facing "import" mentions from the removed import flow (the old +panel Empty text's "import an existing one"): the old Empty string is rewritten (above); two survivors fixed in +place — `settings.tsx` exposure line ("…import and export are copies" → workspace files indexed in place, download/ +export are copies) and the settings search-words list ("import" dropped, workspace/scan/rescan added). No other +user-facing import mentions remain. + +## Tests +New in `test/store.test.mts` (+3): +1. `docsForHome: UNOWNED docs (workspace-scanned and human-made) are visible to EVERY home` — scanned + human doc + seen by every home; another home's owned doc stays private. +2. `scanWorkspace stamps DocMeta.workspace: first path segment under the DEEPEST scan root…` — nested roots, + direct-child-of-root → root dir name, home-root file → home dir name, nested file → first segment. +3. `scanWorkspace rescan BACKFILLS DocMeta.workspace on entries indexed before the stamp` — old index entry picked + up its label on rescan, counted as refreshed once, idempotent after. + +Suite: `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → **146 total / 143 pass / +0 fail / 3 docker-gated skips** (baseline was 143/140/3; +3 new tests), green three consecutive runs. +`npx tsc --noEmit -p extensions/office` exit 0. `node scripts/build-extensions.mjs` exit 0 (gitignored build +outputs refreshed — portal reload is the user's to trigger). + +## Notes / judgment calls +- Deepest-root-wins + root-basename-for-direct-files: both example cases in the brief ("pithagoras" under a + `/root/repos` root; home-root file → home dir name) fall out of this single rule. Documented on `DocMeta.workspace`. +- Workspace label is displayed on the full-page row meta (`· library` for undefined); the panel row layout was kept + as-is per the brief (label still participates in sorting there). diff --git a/tasks/pr307-push.md b/tasks/pr307-push.md new file mode 100644 index 000000000..a6eefec66 --- /dev/null +++ b/tasks/pr307-push.md @@ -0,0 +1,71 @@ +# PR #307 — independent verification, commit & push + +Branch `office/280-pithagoras-office` @ thecodacus/pithagoras. Reports read first: +`tasks/pr307-fixes-core.md`, `tasks/pr307-fixes-docs.md`. All checks run independently of the fix reports. + +## 1) Verification results (all PASS) + +| Check | Command | Result | +| --- | --- | --- | +| Scope | `git status --porcelain` + `git diff --stat` | **PASS** — 18 modified tracked files, ALL under `extensions/office/`; no tracked file outside that dir modified. Untracked noise (`.pi/`, `.reports/`, `.office-contracts.md`, `PLAN-office-280.md`, `tasks/`) left unstaged. Expected new office files present: `src/jszip.ts`, `deps.d.ts` (at extension root), `test/jszip.test.mts`; also `test/live.test.mts`. | +| Tests | `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` | **PASS** — 138 tests: **135 pass / 3 skip (docker-gated) / 0 fail**, 4.4 s (matches reports' expected state). | +| Types | `npx tsc --noEmit -p extensions/office` | **PASS** — "TypeScript: No errors found", exit 0. | +| Build | `node scripts/build-extensions.mjs` | **PASS** — all extensions built, DONE, exit 0; only gitignored outputs (`git check-ignore` confirmed for `server.js` and `web/`). | +| Lazy jszip sanity | grep emitted `extensions/office/server.js` | **PASS** — zero static `import JSZip from "jszip"` / `require("jszip")`; exactly 1 deferred dynamic `import("jszip")`. | + +No breakage fixes were needed; nothing was guessed or patched. + +## 2) Commit + +- SHA: **f4330519db940dc5d57aa79ef24215f76f622104** (parent fee666ce — prior PR tip) +- First line: `Office extension: address code review findings (#307)` +- Body bullets cover all items from both reports: dense-grid parse cap; import realpath/512 MiB/32 MiB guards; CRC-32 validation on read; office_comments 200k-char output cap; presence 15-min TTL; token typ/purpose claims + ticket family marker; rewriteZip case-insensitive dedupe; co-edit docs corrected to real semantics; jszip lazy loader + deps.js wiring (docs pattern); export wording + Deferred entry; single index writer via store.addGrant; wake-text grant wording; README Security notes for host-mode :80. +- Stats: 22 files changed, 606 insertions(+), 109 deletions(-). + +## 3) Staged file list (verified with `git status --porcelain` pre-commit — all under extensions/office/) + +``` +M extensions/office/README.md +M extensions/office/build.mjs +A extensions/office/deps.d.ts (new) +M extensions/office/src/callback.ts +A extensions/office/src/jszip.ts (new) +M extensions/office/src/jwt.ts +M extensions/office/src/live.ts +M extensions/office/src/ooxml.ts +M extensions/office/src/ping.ts +M extensions/office/src/server.ts +M extensions/office/src/store.ts +M extensions/office/src/tickets.ts +M extensions/office/src/tools.ts +M extensions/office/src/web/settings.tsx +A extensions/office/test/jszip.test.mts (new) +M extensions/office/test/jwt-tickets.test.mts +A extensions/office/test/live.test.mts (new) +M extensions/office/test/ooxml.test.mts +M extensions/office/test/ping.test.mts +M extensions/office/test/server.test.mts +M extensions/office/test/store.test.mts +M extensions/office/test/tools.test.mts +``` + +Nothing from `tasks/`, `PLAN-office-280.md`, `.office-contracts.md`, `.pi/`, `.reports/`, or `/tmp` was staged. (`extensions/office/deps.js` was already committed in an earlier PR commit — not re-staged, no changes.) + +## 4) Push + +- Head target from `gh pr view 307`: headRepository = thecodacus/pithagoras (same as remote `origin`, https://github.com/thecodacus/pithagoras), headRefName = `office/280-pithagoras-office`, headRefOid fee666ce. +- Push: `git push origin office/280-pithagoras-office:office/280-pithagoras-office` → **fee666ce..f4330519** (clean fast-forward, no force, single branch). + +## 5) PR tip confirmation + +`gh pr view 307 --json commits` → tip = **f4330519** "Office extension: address code review findings (#307)", headRefOid matches. Note: the very first `gh pr view` right after pushing still showed fee666ce (API propagation lag ~seconds); a re-query confirmed f4330519. Checks will re-run on the new push; not waited on. The previously-failing `server` job is an unrelated extensions/docs flake, expected to be rerun regardless. + +## 6) Cleanup + +- No background processes started by this session (all commands ran synchronously and exited). +- No temp files/dirs created by this session. Existing `/tmp/pithagoras-*` dirs predate it (2026-10-23) and were left alone. +- `git worktree list` → only the main repo (`~/repos/pithagoras f4330519 [office/280-pithagoras-office]`). + +## Anomalies + +- None affecting the push. Only notable observation: GitHub's PR API briefly (seconds) lagged behind the push before reporting the new tip — confirmed resolved by re-query. diff --git a/tasks/pr307-security.md b/tasks/pr307-security.md new file mode 100644 index 000000000..45b63f169 --- /dev/null +++ b/tasks/pr307-security.md @@ -0,0 +1,91 @@ +# PR #307 security audit — pithagoras-office (`extensions/office/`) + +**Branch:** `office/280-pithagoras-office` @ fee666ce (base main) +**Scope:** full security pass over the slice `src/{jwt,tickets,callback,server,ping,container,tools,store,ooxml,live,host,settings}.ts`, `src/web/{api.ts,library.tsx,office-editor.tsx,state.ts}`, `src/client.tsx`, plus the platform seams they rely on (core login gate, `folders.allowed`, pithagoras-docker `sharedNetwork`). + +## Scope + +- JWT HS256 config signing / callback verification (`jwt.ts`) +- HMAC document tickets + loopback ticketed GET (`tickets.ts`, `server.ts` loopback listener) +- DS callback auth, envelope binding, persist path (`callback.ts`) +- Container lifecycle, DS env secrets, shared-network exposure (`container.ts`) +- Authorization: agent tool scoping vs human routes (`tools.ts`, `server.ts` route table) +- Path traversal / zip-slip (`store.ts`, `ooxml.ts`), XSS in the web client, SSRF, secrets/PII in logs, timing/oracles + +## Method + +Read every file in the slice in full; read the platform seams (`core/server/src/auth.ts` cookie flags, `core/server/src/agent-folders.ts` + `packages/extension-api/server.d.ts` for `folders.allowed`, `extensions/docker/src/docker.ts` `sharedNetwork()`) to confirm the trust boundaries each assumption rests on. Cross-checked claims (e.g. "same secret for callbacks/tickets/configs", "single shared login") against code rather than plan text. No test runs; no files modified. + +--- + +## Findings + +### 🔴 [blocking] + +None found. The core crypto/auth design is sound — see "What's good". Everything below is 🟡 or lower. + +### 🟡 [important] + +**S1. In host mode the Document Server's stock nginx listens on ALL interfaces of the real host (not just 127.0.0.1), exposing an unauthenticated community-image web server to LAN/Tailscale.** +- `extensions/office/src/container.ts:210-211` sets `HostConfig.NetworkMode = "host"` when the portal runs directly on a Linux machine; the module header (`container.ts:22-30`) acknowledges "the stock nginx listens on :80 of the whole shared netns". In host mode that is the **host's** netns, and OnlyOffice's stock `listen 80` binds `0.0.0.0:80`. +- **Why it matters:** anything with network reach to the portal machine (LAN, Tailscale — per this deployment both PVE nodes masquerade into Tailnet) can drive the DS web surface: static assets, editor bootstrap, version endpoints — gated by the image's JWT env only for the signed config/callback flows. The PR defers the *container-mode* browser-reach decision to M0, but host-mode 0.0.0.0 exposure is not documented as an accepted choice. This also conflicts with any other service on :80. +- **Fix:** before merge, verify what the pinned `onlyoffice/documentserver:10.0` nginx actually binds and which endpoints are JWT-gated; then either (a) ship a custom nginx conf (named volume at `/etc/nginx/conf.d/ds.conf`) binding `127.0.0.1`, (b) route DS traffic through the session-gated portal, or (c) explicitly document the 0.0.0.0:80 exposure + required firewall rule in the README's deferred section and warn on first start when :80 is not loopback-only. + +**S2. Presence is set at editor-config mint time and cleared unconditionally on DS terminal close — with one shared seat id, a failed open can lock a document to agents forever, or a second tab's close can unlock while a first is still editing.** +- `server.ts:271` (`live.setUser(doc.docId, EDITOR_USER, true)` in `buildEditorConfig`) and `callback.ts` status 2/4 handlers (`clearDoc(docId)` in the persist path and `case 4`). All human sessions share `EDITOR_USER = { id: "portal-user", name: "Portal" }` (`server.ts:57`). +- **Why it matters (two directions):** + - *Stuck lock:* if the config is minted but api.js never loads (S1-style network failure, user closes tab before DS connects), no status-1/2/4 callback ever arrives; `live` state has no TTL (`live.ts` — plain in-memory maps), so the doc shows "editing now" forever and every `office_edit` is refused (`server.ts makeForcesaveHook()` returns false whenever `usersOf(docId)` is non-empty). A human can DoS agent editing of one document by merely opening it once. + - *Early unlock:* with two browser tabs sharing the seat id, a terminal close callback for one tab wipes presence for both; the forcesave hook then resolves true and an agent edit proceeds over the other tab's unsaved buffer — defeating the explicit "never clobber an unsaved buffer" fail-closed design. (Confidence: medium — depends on DS multi-session callback semantics with a duplicated user id, which is not pinned down in code or tests.) +- **Fix:** set presence lazily from the first *DS-confirmed* status-1 (not at mint); add a join-TTL/heartbeat to `live.ts`; and on status 2/4 clear only when the callback's own `users` snapshot (when present) says nobody remains, or track an open-mint counter per doc. At minimum, document the single-seat limitation. + +**S3. One shared secret covers all three token types and lives where any local process of the portal user can read it; no rotation path.** +- `server.ts:ensureJwtSecret` (0600 file in dataDir) → same value in `container.ts:96` (`JWT_SECRET` env) → used by `jwt.ts`, `tickets.ts` and every minted config. +- **Why it matters:** one leak = forge DS callbacks (overwrite any document, bypassing the revision gate, because a forger can read current mtime), mint tickets (read any document for 10 min), or sign editor configs. The blast radius is bounded by "reads as the portal OS user / has docker access" — but that includes *any other extension or tool running in the same portal process/host user*, and there is no rotation mechanism (shortening the file regenerates the secret, `server.ts`, but nothing invalidates already-issued DS env — adopted containers keep the old secret forever, `container.ts` adopt-never-recreate). +- **Fix:** acceptable for a single-user local tool, but: document the trust statement explicitly in README ("any process running as the portal user can read/write office documents"), and add a "rotate secret" setting (new random file + remove/recreate DS container, since env is only set at create). + +**S4. Callback download URLs (`body.url`, `changesurl`) are fetched with no scheme/host validation — SSRF surface tied to the shared secret.** +- `callback.ts:201` `fetchWithRetry(url, …)` fetches whatever the signed envelope says; also unbounded response size (no cap on the downloaded buffer or the bytes persisted). +- **Why it matters:** the URLs come from JWT claims signed with the shared DS secret — normally trustworthy. But combined with S3, anyone holding that secret can make the portal process issue server-side GETs to arbitrary URLs (metadata endpoints, other local services) and write multi-GB bodies into the library. The ooxml reader has deliberate size caps; this path does not. +- **Fix:** validate scheme is `http`/`https`, restrict the host to the DS origin (`state.origin`) or at minimum block link-local/private ranges unless they match the recorded origin; cap downloaded bytes (e.g. 256 MB, matching `ooxml.ts MAX_ENTRY_BYTES`). + +**S5. Wake-on-mention is an unauthenticated prompt-injection + standing access-grant channel into other agents' home chats.** +- `ping.ts:processComment` → `officeWakeMessage` (raw comment text) → queued; drained by `tools.ts:439` via `pi.sendMessage({ customType: "office-comment-ping", content: entry.text, display: true, details }, { deliverAs: "followUp", triggerTurn: true })`. Any co-editor of any doc can type a comment mentioning an agent/project name; that verbatim text then triggers a turn in the target home chat *with* a fresh edit grant on the document and the sentence "You have edit access to this document for this task". +- **Why it matters:** the woken agent's full toolset (not just office tools) sees attacker-authored instructions. This is the feature by design, but nothing marks the text untrusted, scopes the instruction to office tools, or gates cross-project grants — a mention from an ownerless human-created doc can wake any named agent/project. +- **Fix (discuss):** prefix wake text with an explicit "untrusted document content" marker and narrow the instruction ("answer using office_* tools only, nothing else"); consider requiring doc-owner confirmation for mentions that grant to *other* projects' homes. At minimum, record this as an accepted risk in the README deferred section. + +### 🟢 [nit] / 💡 [suggestion] + +**N1. `importFile` follows symlinks — allowed-parent is checked on the logical path only.** +- `server.ts POST /import` checks `portal().folders.allowed(path.dirname(abs))` on the resolved-but-not-realpath'd path; `store.ts:371,407` then `statSync`/`copyFileSync` follow symlinks. A symlink planted inside a workspace folder (e.g. `notes/x.docx → /elsewhere/secret.docx`) passes the check and imports the target. Low value here because agents already have host-filesystem access as the same user, but the containment intent is undercut. **Fix:** `fs.realpathSync` the resolved path and re-run `folders.allowed` on its parent (cheap; closes the class entirely). + +**N2. No size cap on import.** `store.ts:importFile` copies a file of any size into the library; the UI hints "agent workspace folder" only. **Fix:** reject above some ceiling (e.g. 100 MB) or warn — mirrors the callback download concern in S4. + +**N3. Callback logs the full `body.url` verbatim.** `callback.ts:467` (`could not download ${body.url} …`). If a DS build ever puts query tokens in those URLs, they land in portal logs. **Fix:** log host+path only, or omit the URL on failure lines. + +**N4. Failed-comment retry is unbounded.** `ping.ts:handlePersisted` — a comment whose processing throws stays "unseen" and is re-reported (log + attempt) on *every* subsequent persist until it succeeds; a persistent condition (e.g. corrupt index) spams logs every save. **Fix:** keep an attempt counter in the state file or drop after N failures with a single log line. + +**N5. Tickets are replayable within their 10-minute window.** `tickets.ts:verifyTicket` never consumes a ticket; the loopback GET (and, in container mode, every other container in the shared netns — the listener's "loopback" is only private to that netns) can re-fetch any document bytes as long as the token lives. The token also appears in DS-side access logs under the named data volume. Low risk (read-only, one doc, 10 min, no session needed), but **💡 fix if trivial:** consume-on-verify with an in-memory `Set<token>` or bind the ticket to the minted mtime and reject when `store.getDoc().mtimeMs` differs. + +**N6. No user scoping on human-facing routes — correct for v1, worth stating.** Every route (`GET/POST /documents…`, `/import`, `/editor-config`) authorizes by the portal's shared-password login only (confirmed in `core/server/src/auth.ts`: one cookie gate, no per-user identity) and any session can read/edit/delete *all* documents regardless of the `owner`/`grants` chips shown in the UI. The asymmetry with the agent tool scoping is deliberate (`server.ts:55-58` comment), but a future multi-user SSO rollout (the oidc router exists) would silently inherit this; flag in the README deferred list. + +**N7. `verifyJwt` accepts tokens without `exp`.** Only internal minters omit exp (none in production paths), so it's harmless today, but a defensive reject (`exp` required) would shrink the API. 🟢 + +--- + +## What's good + +- **Crypto done right:** `jwt.ts` pins `header.alg === "HS256"` exactly (rejects `none`, RS/algorithm-confusion), recomputes over the *transmitted* segments, and uses `timingSafeEqual`; `tickets.ts` uses constant-time HMAC comparison with a length pre-check. Secret is 256-bit from `crypto.randomBytes`, stored 0600 with explicit chmod (umask-proof), regenerated if truncated. +- **Envelope binding is the standout design:** `callback.ts` takes `key`/`status`/`url` from the *signed* JWT claims, never the unsigned JSON body — a valid token alone cannot POST an arbitrary `{key,status,url}` overwrite. The "verify once, reuse claims" note avoids an expiry-race double-verify as well. +- **Revision gate + double-save memo** (`callback.ts` stale-gate, `store.ts:persistBytes`) implement the promised zero-tolerance mtime check with a coherent treatment of DS's duplicate force-save pass; every failure answers 200/`{"error":1}` per the DS contract. +- **Agent tool authorization is tight:** `tools.ts visibleDoc` scopes list/read/edit/comments to `docsForHome` (owner ∪ grants) and answers out-of-scope ids with the *identical* "no document with that id" text as nonexistent ones — no existence oracle across homes. Duplicate folder names refuse wakes rather than guessing (`ping.ts resolveMention`). +- **Zip handling is defensively capped** (`ooxml.ts`: 256 MiB per-entry, 1 GiB total advertised-uncompressed cap checked before any entry is retained; memory-only reads/writes — no zip-slip surface since parts never touch the filesystem under attacker names). Titles are sanitized before becoming filenames (`store.ts assertSafeTitle`), docIds come from a crypto-random 36^6 space, index writes are atomic tmp+rename. +- **Web client is XSS-clean:** React auto-escaping throughout (no `dangerouslySetInnerHTML`/innerHTML anywhere in the slice); only dynamic URLs are server-built `api.downloadUrl/exportUrl` with `encodeURIComponent`; api.js `src` comes from the server-minted origin, not user input; CSRF is covered platform-wide by the `SameSite=Lax` login cookie (`core/server/src/auth.ts:231`). +- **Loopback listener hygiene:** 405/404 before body reads, 2 MB envelope cap with backpressure, no percent-decoding of the ticket path, unref'd intervals, `closeAllConnections()` on stop. +- `:latest` image tags are refused with a clear reason; adopted containers are never recreated or restarted (no surprise restarts). + +## Open questions + +1. **DS multi-session callbacks with one shared user id** — does OnlyOffice fire status 2/4 per tab while another session is still connected, and do those envelopes carry a `users` snapshot? This determines whether S2's early-unlock direction is exploitable in practice. A test against the pinned image would settle it (deferred to M0 handshake work). +2. **Community image 10.0 nginx bind + JWT coverage** — confirm which endpoints require tokens with `JWT_ENABLED=true` and where `listen 80` binds (drives S1 severity; also whether the client's bare `loadScript(apiJs)` without a `?jwt=` param is accepted by that tag). +3. **Multi-portal / multi-user roadmap** — per-instance secrets, per-user document scoping, and per-comment grant expiry are all deferred; N6/S3 should be revisited when OIDC SSO gains real per-user identities. +4. **Port 80 on the portal host in host mode** — availability (conflicts with existing services) is out of slice but coupled to S1's fix choice. diff --git a/tasks/pr307-tools-verification.md b/tasks/pr307-tools-verification.md new file mode 100644 index 000000000..8bf1d6a6d --- /dev/null +++ b/tasks/pr307-tools-verification.md @@ -0,0 +1,91 @@ +# PR #307 review slice — agent tools + verification runs + +Branch `office/280-pithagoras-office` @ `fee666ce`, reviewed 2026-01 (see dates in repo). Slice: PART A = the five agent tools in `extensions/office/src/tools.ts` and their dependencies (`ooxml.ts`, `store.ts`, `ping.ts`, `live.ts`, `callback.ts` lock model); PART B = test / typecheck / build / docker runs. + +## Scope +- `extensions/office/src/tools.ts` (446 lines, read in full): `office_list`, `office_create`, `office_read`, `office_edit`, `office_comments`, plus the mention-wake drain (`hearOfficePings`). +- Deps actually exercised by those tools: `ooxml.ts` quote-edit engine, `store.ts` (docsForHome/createDoc/persistBytes/getDoc), `ping.ts` (mention grammar, grant persistence, wake queue), `live.ts` (presence/lock model). +- Test corpus `extensions/office/test/*.test.mts` for coverage assessment. + +## Method +- Static read of all listed files; traced gate ordering in `office_edit`, the quote-matching virtual-text algorithm, and the mention regex/grant path end to end. +- Empirical probes via a throwaway `/tmp/office-scratch.mts` (imported the real `editDocx`/`writeZip`; deleted after use — no leftovers): cross-paragraph quote result re-extraction, and 9 mention-regex samples (`@agent`, `@Agent`, `@AGENT`, email, `a@b`, `.@x`, `@project:Finance Team`, `cc @billing.`, Cyrillic `@аgent`). +- Ran the full prescribed PART B verification in `/root/repos/pithagoras`; exact outputs captured below. + +## Findings + +### 🔴 blocking +None found in this slice. + +### 🟡 important + +1. **Mention grants are permanent and there is no revocation path — an incidental mention permanently opens read+edit of a document to another agent home.** + `extensions/office/src/ping.ts:168` (`grants.push({ chatId, commentId, at })`) persists the grant with only an `at` timestamp; expiry is explicitly deferred. `store.docsForHome` (via `doc.owner === homeChatId || doc.grants.some(...)`, `extensions/office/src/store.ts:204-213`) then makes the granted home see the doc in `office_list`, read it fully, AND edit it with `office_edit` — forever. But the wake text says `"You have edit access to this document for this task"` (`ping.ts:71-79`, `officeWakeMessage`) — "for this task", which the persistence does not honor. Worse, any comment that merely *mentions* a matching folder name ("we talked about this with @billing last sprint") grants permanent access even when no task is being requested; there is no revoke tool (owner can see grants via `office_comments` but cannot remove them). + **Why it matters:** grant semantics silently outlive the interaction; over time each doc accumulates standing co-editors across agent homes based on casual text. Deferred decision, so discuss rather than block — but "permanent" contradicts the message handed to the woken agent and is worth a product call before merge. + **Fix:** at minimum (a) change the wake wording to state access is standing until revoked, or (b) keep the deferral explicit in README *and* add an `office_revoke`/owner-side grant removal so permanence is controllable. Longer term: implement per-comment grant expiry (already on the deferred list). + +2. **`office_comments` output is uncapped — inconsistent with `office_read`'s deliberate context guards.** + `extensions/office/src/tools.ts:418-423`: comment bodies are embedded in full into the tool text and also passed whole in `details` (which lands in the portal tool-event store and transcript). `office_read` goes to obvious lengths to bound this (`MAX_READ_CHARS = 200_000` at `tools.ts:293/304/314`, comments capped to 100×500 chars with `commentCount` kept, `tools.ts:317`). A docx with many or long comments (Word allows thousands of comment words each) will blow the agent's context through `office_comments` with no truncation marker at all. + **Fix:** reuse the same caps for `office_comments` — e.g. truncate per-comment text to 500 chars in the listing, cap at 100 comments, and report `commentCount`. + +### 🟢 nit / 💡 suggestion + +3. **🟢 A quote spanning a paragraph break leaves a stray line break inside the replaced region (verified empirically).** + `extensions/office/src/ooxml.ts:315-367` (`applyEdit`): the virtual text joins paragraphs with `"\n"` that has "no XML behind it", so when an op's quote crosses a paragraph boundary, the join survives the rewrite. Verified with the real code: quoting `"end.\nSecond"` and replacing with `"combined"` in two paragraphs extracts afterward as `"First line combined\n line start here."` — i.e. the agent's replacement is split by a newline it didn't ask for, and a quote that fully consumes a paragraph leaves an empty `w:p` (blank line). Exact-substring semantics on extracted text don't round-trip to exact-substring result. Fails loudly? No — this silently produces slightly-off text, which then makes the agent's *next* quote harder (it must re-read and account for the stray break). + **Fix:** either document in the `office_edit` description ("quotes crossing paragraph breaks keep the line break") or have `applyEdit` drop now-empty `w:p` elements / coalesce when a quote consumed an entire paragraph's runs. + +4. **🟢 `@Agent` does not mean "the owner" — case variants of `@agent` silently reroute to folder-name matching (verified).** + `extensions/office/src/ping.ts:112`: the `agent(?![\w-])` alternative is case-sensitive (no `/i`), so `@Agent` / `@AGENT` fall through to the bare-name group. Verified: `@Agent -> ["Agent"]`, resolved by `resolveMention` *case-insensitively against folder display names* (`ping.ts:180-196`). If a folder is named "Agent" (plausible), a human typing `@Agent` intending the doc's owner wakes that other home instead — and gets it a permanent grant (see finding 1). Unicode lookalikes fail safely (`@аgent` → no match, verified). Emails in realistic spelling are safe (`user@example.com`: the `(?<![\w.])` lookbehind blocks the `@` preceded by a word char — verified `a@b`, `.@x`, `user@…` all no-match). + **Fix:** lowercase the owner check too (treat `@agent` case-insensitively as the owner token), or document that only lowercase `@agent` addresses the owner. + +5. **🟢 Non-ASCII folder names can never be mentioned.** + `extensions/office/src/ping.ts:112`: bare names are `[A-Za-z0-9_][\w-]*`, so an agent/project named e.g. "Übersicht" or "café" is unreachable by mention (unresolvable → logged, no wake). Documented-ish ("names are non-whitespace runs, spaces excluded") but the ASCII restriction isn't called out in the module comment. **Fix:** note it as a v1 limitation next to the spaces one; optionally widen to Unicode word chars. + +6. **💡 `office_edit` has no size bound on ops.** + `extensions/office/src/tools.ts:350-357` validates shape but not count or replacement length. Realistically bounded by LLM output, and read-backs are capped — so low risk — but a cap (e.g. ≤ 100 ops, total replacement ≤ N KB) would keep the written `word/document.xml` predictable. + +7. **🟢 forcesave gate keys on presence only (`usersOf`), not the explicit lock flag (speculative).** + `extensions/office/src/tools.ts:371`: `if (loadSettings(...).coEdit && usersOf(doc.docId).length > 0)`. Per the live.ts lock model (`live.ts:16-20`, callback statuses set both presence and an explicit flag), a residual explicit-lock-with-no-presence state (e.g. after a portal restart, or leave-handling edge) would let `office_edit` skip forcesave entirely. Silent clobbering is still impossible — the strict mtime gate in `persistBytes` (`store.ts:440-448`, `st.mtimeMs !== doc.mtimeMs`) makes the human's next save fail loudly — but the *forcesave-first* promise is weaker than it reads under that edge. Low confidence this state occurs often; flagged for awareness. + +### 📚 learning / 🎉 +- The refusal convention is well executed: every failure path routes through `failed()` (`tools.ts:148-162`) into `error:<code> <message>` with *actionable* text — `revision_mismatch` carries the fresh mtime and the re-read instruction twice (gate path at `tools.ts:363` and store path via `currentMtimeMs` injection in `failed()`); `quote_not_found`/`quote_ambiguous`, `session_open` (both flavors: no hook / hook returned non-true, `tools.ts:372-374`), `not_found` for out-of-scope ids are indistinguishable from nonexistent ones (no enumeration leak, `visibleDoc` `tools.ts:219-225`). The suite even pins the contract (`tools.test.mts:382` "every refusal the agent reads starts with error:<code> "). +- Stale-revision guarantee is genuinely layered: gate-first at `tools.ts:365`, re-gate after forcesave at `tools.ts:377`, strict zero-tolerance re-stat in `persistBytes`. Note the agent-side gates use `Math.abs(meta.mtimeMs - base) > 1` (±1 ms, both directions) while `persistBytes` is exact — the final strict gate closes any drift, so "nobody clobbers a newer revision" holds. +- 💡 The "no document with that id" refusal for out-of-scope docs is a deliberate docs-convention choice (`tools.ts:219-224`); fine, just noting an agent guessing ids can't distinguish "doesn't exist" from "not yours" by design. + +## Verification results (PART B, all in /root/repos/pithagoras) + +1. **Unit tests:** `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → + ``` + # tests 123 + # pass 120 + # fail 0 + # cancelled 0 + # skipped 3 + # duration_ms 3577.896591 + ``` + **Discrepancy vs PR body:** the PR claims "120 tests / 117 pass / 3 skip". Reality is **123 total / 120 pass / 3 skip / 0 fail** (off by exactly +3 passing tests — consistent with head commit `fee666ce "fix(office): baseline imported comments at import time"` adding tests). The PR body's verification numbers are stale; zero failures confirmed. + +2. **TypeScript:** repo root has **no tsconfig.json** (`npx tsc --noEmit -p .` → `error TS5057: Cannot find a tsconfig.json file at the specified directory: '.'`). As instructed, tried the extension project: `npx tsc --noEmit -p extensions/office` → **`TypeScript: No errors found`, exit 0** (strict mode, `include: ["src"]`). Clean as claimed — with the caveat that test files are not covered by any tsconfig include. + +3. **Build:** `node scripts/build-extensions.mjs` → for the relevant line: + ``` + pithagoras-office: built web/ and server.js + ``` + (all six other extensions also built, exit 0). Build artifacts land in the repo tree (`extensions/office/web/*`, `extensions/office/server.js`) but are **gitignored** — `git check-ignore` confirms both paths. Post-build `git status --short`: only the four pre-existing untracked items from checkout time (`.office-contracts.md`, `.pi/`, `.reports/`, `PLAN-office-280.md`, plus this review's `tasks/`) — **no tracked-file pollution**. + +4. **Docker:** `docker info` → daemon present and answering. `docker images --format '{{.Repository}}' | grep -i onlyoffice` → **no DS image locally**, so per instructions no pull was attempted; the 3 docker-gated integration tests remain skipped. Their exact names (from `test/integration.test.mts`, gate at lines 55-60, requires `OFFICE_IT_DOCKER=1` + operator-started container): + - "ticketed GET on the real loopback listener serves a >1 MB document byte-for-byte and rejects an expired ticket" + - "synthetic status-2 callback WITHOUT any session cookie: persist happened (bytes changed, index bumped, history zip written) and the lock cleared" + - "office_edit with a live session: the forcesave hook is AWAITED before the edit persists; a revision bump during the hook surfaces as revision_mismatch" + +## Coverage gaps +- **Quote editing across paragraphs** (stray-break behavior of finding 3), the "covers no text runs (paragraph breaks only)" branch (`ooxml.ts:365`), and literal CRLF inside `w:t` content are untested — `test/ooxml.test.mts` covers single-run, multi-run, sequential-ops, ambiguous/not-found only. +- **`office_read` truncation paths** (200k text cap, 100×500 detail cap) and the xlsx `truncatedSheets` notice: no test drives a >200k doc or an out-of-grid worksheet through the *tool* surface. +- **`office_comments`**: only the happy path is tested (`tools.test.mts:350`) — its unbounded output (finding 2) is therefore unpinned. +- **Mention grammar edges** (case variants, Unicode names/lookalikes, punctuation-heavy `@project:` targets): `test/ping.test.mts` covers case-insensitivity, unmatched, ambiguous, dedupe, baselines — not the ASCII-only/case-sensitive-literal corners above. +- The 3 docker-gated ITs above remain unverified here (no OnlyOffice image; not pulled). Notably IT #3 (forcesave ordering) has a unit twin (`tools.test.mts:294`) so coverage exists, just not against a real DS loopback. + +## Open questions +- Should the PR body's verification numbers be updated to 123/120 before merge, or was "120 tests" meant as "120 pass" (in which case only "117 pass" is wrong)? +- Finding 1: is permanent grant semantics an accepted v1 deferral (it's on the README deferred list) — but should the wake message at least stop promising "for this task"? +- Does any root-level CI typecheck/build gate exist for the repo that this extension expects? (`tsc -p .` has nothing to check at root.) diff --git a/tasks/pr307-triage-architecture.md b/tasks/pr307-triage-architecture.md new file mode 100644 index 000000000..c8db9683d --- /dev/null +++ b/tasks/pr307-triage-architecture.md @@ -0,0 +1,108 @@ +# PR #307 triage + architecture review — extensions/office ("pithagoras-office") + +Branch `office/280-pithagoras-office` @ fee666ce, base main. 44 files, +8703/-1. + +## Scope + +Files assigned to this slice: `extensions/office/{README.md,build.mjs,deps.js,package.json,package-lock.json,tsconfig.json,.gitignore}`, `src/{settings.ts,host.ts,portal.tsx,office-ui.ts}`, the 1-line change in `core/server/test/extension-ui.test.mjs`, plus a skim of every other `src/*` and `test/*` file to map module boundaries. Reference: `architecture-review-guide.md`; comparison against sibling extensions (`docker`, `files`, `kanban`, `canvas`, `docs`). + +## Method + +- Full `git diff main...HEAD --stat` + read of all 13 server/client TS modules, README, build/packaging files, tsconfig, and the core test hunk. +- Structural diff vs 4–5 sibling extensions (layout, build.mjs, deps.js precedent in `docs`, cross-extension imports, tsconfig identity). +- Dependency-direction mapping across settings/host/server/callback/container/ping/store/tools/web; verification of wiring claims (imports actually used, routes registered, hooks attached) with grep + reading the built `server.js` to confirm what esbuild externalizes. +- Behavior-vs-docs cross-check: PR body / README / UI copy vs actual code paths (co-edit gate, forcesave hook, import/export endpoints, naming registration). + +## Findings + +### 🔴 [blocking] + +None found in this slice. The concurrency/credential design (revision gate, signed callback envelope, loopback scoping) is sound; the issues below are documentation/wiring/boundary problems, not merge-stoppers as of this commit. + +### 🟡 [important] + +**1. Co-editing semantics are described backwards in the UI and README — the headline feature's mental model is inverted.** +- `src/web/settings.tsx:109` (coEdit toggle detail): "While **on**, an agent edit of a document somebody has open **proceeds** under the revision gate with a force-save request first; while **off**, it is **refused until the editor is closed**." +- Actual v1 behavior: `src/tools.ts:371` only enters the forcesave branch `if (loadSettings(rootDir()).coEdit && usersOf(doc.docId).length > 0)`, and the wired hook (`src/server.ts:140` `makeForcesaveHook`) **always returns false while a session is open** — so with coEdit ON, agent edits are *refused* (`error:session_open`) for as long as anyone has the editor open; with coEdit OFF, the branch is skipped entirely and agent edits *proceed* under the bare mtime gate even with a human session open. +- `README.md:11` repeats the same overclaim ("An open editor session does not block an agent edit — the extension force-saves the session first"). The test suite itself asserts the refusal (`test/server.test.mts:557–572`), i.e. code and docs disagree. +- Why it matters: a user who reads the toggle believes OFF is "the safe setting"; turning it off actually *removes* the only protection for an open human buffer (agent writes behind DS's back, then the human's save hits the stale-key gate at `src/callback.ts` persist path and fails in the editor). And users with defaults never see the advertised "co-editing". +- Fix: rewrite `settings.tsx:109` to state v1 reality ("with an editor session open, agent edits are refused until it closes or saves; a force-save flush is planned for M0") and fix `README.md:11`; add "browser-side forcesave-first (v1 refuses while a session is open)" to the README *Deferred* list. If the intended v1 semantics were instead "proceed after confirmed flush", that would be a functional gap in `makeForcesaveHook`, not a copy bug — worth one explicit sentence from the author on which is intended. + +**2. `deps.js` is committed but dead — and the runtime-dependency guarantee it exists to provide is never wired.** +- Nothing imports `extensions/office/deps.js`: not `src/*`, not `test/*`, not `build.mjs`. The build duplicates its missing-deps check inline (`build.mjs:24–27`), mirroring what the docs extension does by *importing* it (`extensions/docs/build.mjs:15`). +- Consequence: the scenario the file's own header promises to handle ("a folder that comes without them (a checkout the portal runs from, a copy made by hand) has them installed here, once") does not exist in office — `ensureDependencies` is never called at server boot. The built half imports jszip as an external (`server.js:350` `import JSZip from "jszip"`; esbuild leaves packages external), so a package copy installed by path or hand-copied *without* its `node_modules/` fails to load the whole server half at import time with `ERR_MODULE_NOT_FOUND`. In the primary shipped flow (`extensions/office/` checkout, `npm ci` run by build.mjs) it works — hence not blocking. +- Fix (pick one): (a) follow the docs pattern — import `missingDependencies` in `build.mjs` instead of duplicating, and call `ensureDependencies(portal().log)` during boot before any template write; or (b) cleaner: bundle jszip into `server.js` (it is pure JS, no native bits), which removes the runtime `node_modules` requirement entirely and lets `deps.js` be deleted. Either way, as-is it is a boat anchor plus a missing safety net. + +**3. "Export to chat folders" is claimed in the PR body/README but does not exist — and it is missing from the honest deferred list.** +- README *What it is* (line ~7): "import/export .docx/.xlsx to chat folders". `src/server.ts:366` `POST /import` genuinely imports *from* an agent workspace folder (gated by `portal().folders.allowed(parent)`). But `src/server.ts:384–409` `GET /documents/:id/export` only copies the bytes to a temp dir, serves them as an attachment, and deletes the temp dir — nothing is ever written into any chat/agent folder, and there is no other export route. The UI backs this up (`src/web/library.tsx:155–158`: "Download" and "Export a fresh copy" are both plain `<a href>` downloads). +- Why it matters: the PR body and README sell a v1 feature that isn't shipped, and unlike xlsx comments / refresh handshake / M0 reach, this absence is not on the *Deferred (honest list)*. +- Fix: implement `POST /documents/:id/export {path}` mirroring import in reverse (target parent must satisfy `folders.allowed`), or add "export to chat folders (v1 exports are downloads)" to Deferred and fix both texts. + +**4. `ping.ts` keeps a second, inline writer for `index.json`, bypassing the module that owns it.** +- `src/ping.ts:158–173` `addGrant()` re-implements read-index → mutate doc.grants → atomic tmp+rename (`fs.renameSync(tmp, p)` at :171) — a duplicate of `src/store.ts:61` `writeIndex`, while store.ts's header declares index.json "the single source of truth" and exposes no grant mutation. The code concedes it: "If grant mutations are centralized in store.ts later, this helper goes away." +- Why it matters: the index has a `version: 1` field; any schema change (e.g. adding per-comment grant expiry, which is on the deferred list) now has to be remembered in two hand-rolled writers, and both use their own shape assumptions (`raw.docs`, `doc.grants ??= []`). Single-process sync read-modify-write means no live race today — this is an encapsulation/evolvability issue, not a correctness one. +- Fix: add `store.grantAccess(docId, chatId, commentId): boolean` (dedupe by `(chatId, commentId)`, `notFound` if the doc vanished between resolve and grant) and delete the inline writer in ping.ts. + +### 🟢 [nit] / 💡 [suggestion] + +**5. 💡 Unpinned image tags slip past the pin check → implicit `:latest`.** +- `src/server.ts:435` refuses `/:(latest|stable)$/i` or `@latest`, but a tag with *no* tag at all (e.g. `onlyoffice/documentserver`) passes and pulls `:latest` implicitly — precisely the "moved image breaks co-editing keys" scenario the refusal exists for. Suggestion: require an explicit pin — reject unless the tag contains a digest (`@sha256:`) or an explicit `:<tag>` after the last `/`. (Confidence: high on the regex gap; low that it matters in practice — the UI pre-fills the pinned default.) + +**6. 🟢 `writeAtomic` (tmp-file + rename) is copy-pasted in four places.** +- `src/settings.ts:51`, `src/callback.ts:212`, `src/container.ts:102`, and inline twice in `src/ping.ts` (:148/:171), plus a third variant inside `store.writeIndex` (`src/store.ts:61–69`). All five are the same 5-line idiom with the same `.${name}.${pid}.tmp` convention. Suggest one shared internal module (e.g. `src/fsutil.ts`) — cheap now, and it keeps the tmp naming consistent if any of them ever needs a change. (Not a blocker; the copies are currently identical in behavior.) + +**7. 🟢 No dedicated unit test for `src/live.ts`; no browser-level tests.** +- `live.ts` carries the most subtle state model in the extension (explicit lock flag vs presence-as-lock, terminal-close authority, meta resolver) but is only exercised through callback/server/tools/integration tests (no `test/live.test.mts`). Suggest a small dedicated file for the edge cases (`setUser` keep-original-`at`, unlock never hides presence, `pollState.pendingRev` with/without resolver). Also: sibling extensions ship browser tests (`extensions/docker/test/browser/docker.spec.mts`) while office has none — the once-per-page `api.js` load + DocEditor lifecycle in `src/web/office-editor.tsx` (script cache that *keeps* a rejected promise on failure, :28–46) is exactly the kind of thing a page-level test would pin down. + +**8. 🟢 Client/server type duplication drift risk.** +- `src/web/api.ts` re-declares `OfficeDoc`/`OfficeContainer` hand-mirrored from `store.DocMeta`/`container.status` (expected — separate bundle, and it already omits `imported`/`importedBaseline`), but there is no shared type source of truth. Fine for v1; worth a comment or generated snippet if the meta grows fields (grant expiry would be the first). + +**9. 📚 Verified non-issues from the triage brief.** +- *Naming:* `office/` dir vs `pithagoras-office` name is **not a mismatch** — every shipped extension follows `<dir>` = package name minus the `pithagoras-` prefix (docker→pithagoras-docker, files→pithagoras-files, …), and the core test hunk (`core/server/test/extension-ui.test.mjs:216`) asserts exactly that mapping (`spec.endsWith(extensions/<name without prefix>)`). Registration, log prefixes, `DS_NAME = "pithagoras-office-ds"`, routes (`/x/pithagoras-office`, `/api/x/pithagoras-office`) all key off the package name. Consistent. +- *Cross-extension imports:* `src/host.ts:2` / `src/container.ts:45` importing types from `../../docker/src/api` matches existing precedent (browser, voice, understory do the same), and both are type-only. No runtime cross-extension coupling. +- *tsconfig/packaging:* byte-identical tsconfig to `extensions/docker` (`strict`, `noEmit`, ES2022); build via repo `@pithagoras/extension-api/build` like siblings; `.gitignore` matches the kanban variant exactly. No divergence. +- *settings schema:* opt-in default (`enabled: false`), pinned tag default, field-by-field merge with silent fallback on corruption (documented at the module header of `src/settings.ts`) — reasonable for a local trusted file; only asymmetry is #5/#7-of-load (no format validation on load, only in `POST /settings`). + +### Module-boundary map (what I checked) + +``` +host.ts ← composition seam: ServerPortal, dataDir, docker() (type-only from extensions/docker) +settings.ts → host(dataDir) [pure-ish, fs over root] +store.ts → host, ooxml(parseDocx, import baseline), jszip(templates) [owns index.json + docs/ layout] +ooxml.ts → node:zlib only [pure over buffers] +jwt.ts/tickets.ts → node:crypto only [pure; shared secret string owned by caller] +live.ts → (nothing) [pure process memory + injected resolver] +callback.ts → host, jwt, store, live [deliberately NOT container — keeps CallbackState local; documented] +ping.ts → host, store(getDoc), ooxml, callback(onPersisted) [+ inline index writer — finding #4] +tools.ts → host, store, ooxml, live, settings, ping(drainWakeQueue) +container.ts → host(settings via loadSettings, docker type-only), fs (ds.json, user-stopped marker) +server.ts → all of the above [composition root: wires seams once, owns loopback listener + routes] +web/* → portal.tsx(slot/api seam), api.ts(hand-mirrored types) — client bundle, no server imports +``` +Direction is clean (leaves are fs/portal-free; only server.ts knows everything); the two boundary blurs are findings #4 (ping→index.json) and the minor `store→ooxml` import for import-time comment baselines (`src/store.ts:19,384`) — the latter I accept: it's documented, one function, and moving it up would push docx internals into the route layer instead. + +No circular dependencies; no god objects except `server.ts` (561 lines), which is defensible as the composition root + HTTP surface for a single-extension process. + +### Test strategy (architecture view) + +120 unit tests across 10 files + 3 docker-gated integration tests (`test/integration.test.mts`, gated on `OFFICE_IT_DOCKER=1` + `docker info` probe, skipping with explicit reasons). Coverage is strong where the logic is dense: callback status machine incl. double-save memo & stale-key refusal (598 lines), container lifecycle with a fake Docker (343), all UI routes + boot/loopback against a fake portal (577), tools refusals/revision gate (417), ping mention grammar/baselining (301), store ops (304), ooxml zip codec cross-validated vs jszip + committed fixtures. Gaps: `live.ts` directly (#7), browser half end-to-end (#7), and the *real* DS save loop stays human-operated by design — honestly stated in the integration header ("not invented here"), which I accept for v1. + +### README honesty check + +*Deferred (honest list)* matches code well: xlsx comments (M4) ✓, refresh handshake/M0 reach + option-a recorded origin with loud `container:<id>` refusal (`src/server.ts:246` → 422 JSON) ✓, portal proxy for DS traffic ✓, `.pptx`/`office_comment_add`/grant expiry/postMessage badges as M4 stretch ✓. **Not** on the list but absent: "export to chat folders" (#3) and the forcesave-first flush that v1 cannot perform (#1). Those two are the honest-list gaps. + +## What's good + +- 🎉 Composition discipline is excellent for a new extension: `server.ts` wires every seam exactly once at boot; every stateful module takes injectable seams with lazy defaults so 120 unit tests never touch `portal()` or the Docker socket; module headers read like contracts (and the comments document *divergences from the plan*, e.g. no `HostConfig.Memory` cap, `container:` origin marker, zip re-compression deviation — rare and valuable). +- 🎉 The callback handler's envelope-from-signed-claims design (`src/callback.ts` "Envelope binding (security)") closes what would otherwise be a trivial overwrite primitive: with a valid token alone an attacker can still only persist what DS itself signed. Status-table fidelity (200 + `{"error":1}` on persist failure) is implemented exactly as documented in the module header. +- 🎉 Deliberate dependency inversion where it counts: `callback.ts` keeps a local `CallbackState` type instead of importing the Docker half; `live.ts` stays fs/portal-free with an injected meta resolver; `ooxml/jwt/tickets` are pure and trivially testable. +- 🎉 The core change is surgically minimal — one asserted array entry in `core/server/test/extension-ui.test.mjs`, no core behavior touched. +- 🎉 Store discipline: atomic index writes, mtime-as-revision with a strict write-time gate, safe-title validation (`assertSafeTitle`), 7-day trash sweep, adopt-never-recreate container policy with a durable `user-stopped` marker file. + +## Open questions + +1. **Intended v1 co-edit semantics** — is "agent refused while any human session is open" (what ships) the accepted reading of plan 5.7 tier-1, or should `makeForcesaveHook` reach the browser for a real forcesave? Determines whether finding #1 is copy-only or also functional. +2. **Human-UI scoping** — all `/documents*` routes are unscoped: any authenticated portal session can list/download/delete *any* document (owner/grants scope only the agent tools and chat panel). Fine for a single-user homelab, but worth an explicit sentence in the README; multi-user would need per-route checks. +3. **Ticket replay** — `GET /doc/<ticket>` tickets are HMAC'd + 10-min TTL but not consumed-once; any local process can replay one within its window (mitigated by 127.0.0.1-only binding). Acceptable for v1? +4. **Presence identity** — `EDITOR_USER = { id: "portal-user", name: "Portal" }` collapses all humans into one seat, so two real co-editors show as one and can't be distinguished in lock badges. Known limitation (commented) — confirm it's acceptable. +5. **History retention** — `history/<docId>/<rev>.zip` archives are written per persist with no pruning anywhere (only removed when the doc itself hits the 7-day trash sweep). Unbounded growth is presumably fine for v1; worth a note or a cap. diff --git a/tasks/pr307-workspace-library.md b/tasks/pr307-workspace-library.md new file mode 100644 index 000000000..f41cd610d --- /dev/null +++ b/tasks/pr307-workspace-library.md @@ -0,0 +1,61 @@ +# PR #307 — office: create-bug diagnosis + workspace library (scan/index, sortable) + +Branch `office/280-pithagoras-office`, work lands on top of pushed `f4330519` + the uncommitted co-editor-model changes. All numbers below re-verified at the finish line. + +## (a) Task 1 — "cannot create a new document": root cause, proven with logs + +**Root cause: the live portal is serving a STALE pre-12:18 build of the office extension.** The running process is `node server/dist/index.js` from `/root/pithagoras`, pid **2115289** (started Oct 24 10:19), which loaded the office half at **Oct 24 11:47** from the pi package path `../../repos/pithagoras/extensions/office` (`/root/.pi/agent/settings.json`, added that same morning per the `settings.json.bak-20261024-114641` diff). The rebuilt bundles in-tree are newer than the loaded code: + +* Journal (systemd, process 2115289 — the exact lines that proved it): + ``` + Oct 24 11:47:51 lxcpithagoras node[2115289]: [extension pithagoras-office] server half started + Oct 24 11:56:52 lxcpithagoras node[2115289]: [extension pithagoras-office] pithagoras-office: pithagoras-office-ds -- document server started + Oct 24 11:58:03 lxcpithagoras node[2115289]: [extension pithagoras-office] pithagoras-office: pithagoras-office settings -- enabled=true imageTag=onlyoffice/documentserver:10.0 coEdit=true + ``` + **No further "server half started" line exists** — the extension has not restarted since 11:47, i.e. it runs code from before the noon rebuild. +* Proof the loaded code is old: `journalctl` shows the live audit at 11:58 **accepted a `coEdit` settings key**, and `/var/lib/pithagoras/extension-data/pithagoras-office/settings.json` still contains `"coEdit": true`. Only the pre-co-editor-model bundle accepted that key — current source and the rebuilt bundles contain no `coEdit` at all (`grep -c coEdit server.js web/client.js` = 0 in both, built 12:18:59). +* The old code also swallowed the create failure on the UI side (no log line is left for it — there is NO error entry anywhere around `POST /documents`), so the click looks dead. +* Dependencies were never the missing piece at boot time: `node_modules/jszip` in the office folder exists with mtime Oct 23 16:28, i.e. BEFORE the 11:47 half start; `import("jszip")` from the office dir resolves fine (`typeof m.default === function`). +* **The current-tree creation path is proven good end-to-end**: direct `store.createDoc` probe on a tmp dataDir → 201, docx 1740 bytes; and a full express round-trip against the freshly built `server.js` with a stub portal → `POST /documents` **201** for both `word` and `cell`, `GET /state` 200. So: after whatever reloads the extension (a later deploy/restart — NOT done here, per instructions), creation works; until then the button hits dead/stale code. + +**Robustness fix implemented so the failure mode can never be a dead click again:** +* `POST /documents` in `src/server.ts` now catches jszip/dependency load failures (`isDependencyError()` walks the `.cause` chain for `ERR_MODULE_NOT_FOUND` naming jszip or an install message) and answers **503 `{ error: "...jszip dependency is still installing... retry in a few seconds.", code: "dependencies_pending" }`**; every OTHER createDoc failure now gets a `logLine(`POST /documents failed: ...`)` before the coded response — route errors previously left no trace at all, which is exactly why this incident had no log to read. +* The UI already surfaces it verbatim: core's `json()` throws `ApiError(body.error)` (verified `core/web/src/api.ts`), and `NewDocument` renders it in an `ErrorBanner`. + +## (b) Task 2 — scan roots: which platform API, and what resolves here + +**API: `portal().folders.list(): AgentFolder[]`** — declared in `packages/extension-api/server.d.ts` ("Every agent's home, every project, and every folder a chat of any kind works in, each once"), implemented by core at `core/server/src/agent-folders.ts:listFolders()` (wired into the extension context at `core/server/src/extension-host.ts:551`, `list: () => listFolders()`). No paths are hardcoded anywhere in the office code — both the boot scan and `POST /documents/rescan` call `portal().folders.list().map((f) => f.path)` fresh. + +On THIS machine (live portal pid 2115289, `WORKSPACE_ROOT=/root/repos` per `/proc/<pid>/environ`; core default would be `/workspaces`): the resolved roots are `/root/repos` itself plus every agent/project/chat folder beneath it (`listProjects(workspaceRoot())` in agent-folders.ts). The office's own data dir (`/var/lib/pithagoras/extension-data/pithagoras-office`) is OUTSIDE that tree, but the scanner still defensively skips its own realpath if it ever were nested. `find /root/repos -name '*.docx' -o -name '*.xlsx'` currently returns nothing, so a live rescan today would honestly index 0 files. + +## (c) Files changed and why (all under extensions/office/** unless noted) + +| File | Why | +|---|---| +| `src/store.ts` | Added `DocMeta.path?: string` (in-place identity for scanned docs, realpath); `docPath()` prefers it; NEW `scanWorkspace(roots, root)` + `indexOne()` — sync bounded walk (skips own dataDir, `.trash`, hidden dirs, `node_modules`; case-insensitive `*.docx/*.xlsx`; realpath identity so symlinked duplicates index once), 512 MiB cap, unsafe-title skip, `importedBaseline` stamped AT INDEX TIME for docx ≤32 MiB (pre-existing @mentions never wake at index time); refreshes mtimeMs/sizeBytes on rescan (only real changes count as `updated`). `renameDoc` renames inside the file's own dir (workspace files rename in place, library files unchanged behavior); `deleteDoc` moves a scanned doc's REAL file to `.trash` (7-day sweep) and treats an already-vanished file as a clean index drop instead of ENOENT; `importFile` kept (ping.test exercises its copy-into-library baseline flow) but marked superseded. | +| `src/server.ts` | Boot now runs `scanWorkspace(folders.list() paths)` AFTER `settleKept` (prune first, so vanished files aren't re-added) and logs counts; NEW route `POST /documents/rescan → {added, updated}` (reads folders live per request); DELETED `POST /import`; Task-1 fix: 503 `dependencies_pending` + `logLine` for create failures; `fileFor()` honors `DocMeta.path` so download/export serve the real workspace file. | +| `src/web/api.ts` | Dropped `api.importFile`; added `api.rescanLibrary()`; documented that core's `ApiError(body.error)` makes every refusal (incl. the 503) land in the UI verbatim. | +| `src/web/library.tsx` | Removed the `ImportDocuments` form and all import copy (header comment, PageHeader description, Empty state); added sort controls — `portal.ui.Select` (name/modified/size/type) + direction toggle, persisted to `localStorage` (`po-sort`), default modified-descending, pure client-side over the polled payload (`useMemo`); added a Rescan workspace ghost button that says its counts in a hint line. Row layout untouched. | +| `style.css` | `.po-listhead`, `.po-sortbar`, `.po-rescan` (flex rows consistent with existing `.po-form`/`.po-hint`). | +| `src/ping.ts` | Comment-only: baseline stampers are now `importFile / scanWorkspace`. | +| `test/store.test.mts` | +5 tests: nested/case-insensitive discovery + skip rules; own-dir (docs/ and .trash) never indexed even when passed as a root; rescan refresh semantics + in-place mtime-gated persist on a scanned doc; oversized (sparse 513 MiB)/unsafe-title/title-collision are quiet skips; deleteDoc moves the real file to trash, vanished-file delete is a clean drop, `pruneMissingDocs` agrees. | +| `test/server.test.mts` | Replaced both `/import` route tests with: rescan indexes in place (no copy into docs/), skips hidden/node_modules/.txt, idempotent second run, refresh after out-of-band write, `/state` payload carries `mtimeMs`/`sizeBytes`/`path`, export serves the real file's bytes, and `POST /import` is **unregistered** (`assert.rejects ... no registered POST route for /import`); plus a rescan baseline-stamping test (real docx with a pre-existing comment → `importedBaseline ["1"]`). | +| `README.md` | Library bullet + Setup step 2 rewritten: no manual import — automatic boot scan (+ Rescan button), in-place docs, deleting a scanned doc moves the real file to the 7-day trash. | +| `.office-contracts.md` (repo root) | Endpoint docs updated: `POST /import` gone, `POST /documents/rescan` added, `/state` docs now carry mtimeMs/sizeBytes (+path), store interface gains `scanWorkspace`, deleteDoc/prune semantics for external paths. | +| `PLAN-office-280.md` (repo root) | Supersession note at §2 where import is described as first-class: manual import replaced by the `folders.list()` scan; export-to-chat-folders stays deferred. | + +## (d) Finish line (all at once, re-run after every edit) + +* Tests: `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → **140 pass / 0 fail / 3 skipped** (the 3 docker-gated integration skips are the pre-existing ones; baseline was 135/0/3). +* `npx tsc --noEmit -p extensions/office` → **exit 0, no errors**. +* `node scripts/build-extensions.mjs` → **DONE, exit 0** (office built web/ + server.js among all packages). +* `git status`: modified files exactly `extensions/office/{README.md, src/live.ts, src/ping.ts, src/server.ts, src/settings.ts, src/store.ts, src/tools.ts, src/web/api.ts, src/web/library.tsx, src/web/settings.tsx, style.css, test/container.test.mts, test/integration.test.mts, test/server.test.mts, test/settings.test.mts, test/store.test.mts, test/tools.test.mts}` (the non-office-hunk ones are the co-editor's uncommitted changes, untouched) + untracked `.office-contracts.md`, `PLAN-office-280.md`, `tasks/` (`.pi/`, `.reports/` predate this session). + +## (e) Deliberately NOT done + +* **No redeploy/restart of the live portal** — the stale bundle is only fixed by a reload/deploy, which the task assigned to a later push agent; nothing was killed or rewritten outside its log files (logs read-only, via `journalctl`/`/proc`). +* No changes under `core/` or other extensions (read for discovery only). +* Kept `store.importFile` and its 4 store tests instead of deleting them: ping.test's first-save wake semantics are the product behavior and it exercises the copy flow cheaply; the route/UI surface is gone, which is what the decision required. +* The boot scan runs once at startup (plus manual Rescan) — there is no interval-based rescan; a `folders.removed`-triggered unindex also does not exist yet (a workspace file deleted out-of-band simply prunes itself at the next settleKept/boot). +* `DocMeta.path` deliberately appears in the human `/state` payload but never in agent-facing tool output: tools scope through `docsForHome`, and every scanned doc is `owner:null`, so path-carrying docs are invisible to `office_*` tools by construction. +* Did not add xlsx baseline stamping (docx only) — consistent with the pre-existing importFile behavior, and xlsx comment wakes are already an M4 item. diff --git a/tasks/summary.md b/tasks/summary.md new file mode 100644 index 000000000..66cbc1d75 --- /dev/null +++ b/tasks/summary.md @@ -0,0 +1,151 @@ +# PR #307 Consolidated Review Summary + +PR: thecodacus/pithagoras #307 "Office documents in the portal: OnlyOffice Document Server extension" (author Piggidragon, OPEN, MERGEABLE) +Branch office/280-pithagoras-office @ fee666ce vs main tip 138738f5 — +8703/-1 across 44 files; everything under extensions/office/ plus one line in core/server/test/extension-ui.test.mjs. Per PLAN-office-280.md / issue #280. + +## Overall verdict + +🔄 **Request Changes** — one merge-relevant 🔴 blocker in PR code (dense-grid xlsx parse), several 🟡 to address before merge, and a ready-to-post answer to the one unresolved CodeRabbit comment. The failing CI test is an unrelated PRE-EXISTING bug in extensions/docs: rerun CI + file a separate issue; it does not gate this PR. + +## 🔴 Blocking (PR code) + +1. **Dense-grid `parseXlsx` scales with cell coordinates, not file size** (tasks/pr307-ooxml.md). Measured 1671-byte xlsx → +523 MB heap / ~18 s CPU; multiple GB possible for a sparse workbook with high row/col refs. No per-file parse guard; reachable from project-chat agents (office_read/edit paths parse inline, extensions/office/src/tools.ts:190). `truncatedSheets` plumbing already exists as the fix slot-in. Fix + follow-up wall-clock guard for huge legit workbooks (open question Q2 in that file). + +## 🟡 Should address before merge + +**Architecture / docs-vs-behavior (tasks/pr307-triage-architecture.md, 4 items):** +1. Co-editing semantics INVERTED in docs: web/settings.tsx:109 and README.md:11 claim ON=proceeds/forcesave, OFF=refused; actual v1 = tools.ts:371 forcesave branch requires coEdit&&usersOf>0 AND server.ts:140 makeForcesaveHook ALWAYS returns false while a session is open ⇒ ON = agent edits refused (error:session_open) until the editor closes; OFF = proceeds under the bare mtime gate with a human session open. test/server.test.mts:557-572 asserts the refusal. Fix docs (or flip behavior). +2. extensions/office/deps.js committed but DEAD — nothing imports it, ensureDependencies never called at boot; built server.js:350 keeps `import JSZip from "jszip"` external ⇒ a hand-copied install without node_modules fails ERR_MODULE_NOT_FOUND. Fix: import missingDependencies in build.mjs (docs pattern) + call ensureDependencies at boot, OR bundle jszip and delete deps.js. +3. "Export to chat folders" is claimed in PR body/README but DOES NOT EXIST — server.ts:384-409 export = copy to temp dir, serve attachment, delete; web/library.tsx:155-158 both plain downloads. Also absent from the honest Deferred list. +4. ping.ts:158-173 addGrant re-implements index.json read-modify-write+rename, bypassing store.ts writeIndex (:61) — a second hand-rolled writer over a versioned schema. + +**Security (tasks/pr307-security.md):** +- S1: host-mode DS runs nginx bound 0.0.0.0:80 (container.ts:210-211 NetworkMode="host"), exposing the community image's web surface to LAN/Tailnet — an undocumented accepted choice. Fix options: verify pinned onlyoffice/documentserver:10.0 bindings; custom nginx conf binding 127.0.0.1 via named volume /etc/nginx/conf.d/ds.conf; or document + firewall. +- S2 (medium confidence): presence is minted at config time (server.ts:271 live.setUser(docId, EDITOR_USER={id:"portal-user",name:"Portal"}, true)) and cleared unconditionally on terminal close; one shared seat id ⇒ a failed open = stuck lock forever (no TTL in live.ts) DoSes agent edits, and a second tab's close unlocks the first tab's unsaved buffer. +- S3: one shared secret covers all three token types (full detail in file). + +**OOXML (tasks/pr307-ooxml.md):** no CRC-32 validation on read — corrupt archives are silently re-persisted with their original wrong CRC (open question Q1: import-reject vs accept-warn); nested w:p (text boxes / mc:AlternateContent) corrupts extraction and mis-targets quote edits, doc comment describes different behavior. 🟢 notes: committed fixtures embed wall-clock timestamps; case-only name mismatches in rewriteZip create shadow duplicate parts. + +**Concurrency (tasks/pr307-callback-concurrency.md):** no 🔴. Multi-MB docx sync parse blocks the event loop for tens of ms per persist — accepted at current scale, with escape valve "move diff/wake to a queue if docs grow". Same stance as CR-latest below: consistent, not a conflict. + +**Agent tools (tasks/pr307-tools-verification.md):** 🟡 mention grants are permanent with no revocation while the wake text promises access only "for this task" (extensions/office/src/ping.ts:168); 🟡 office_comments output is unbounded unlike office_read (extensions/office/src/tools.ts:418). 🟢 cross-paragraph quote edit leaves a stray line break. + +## CI failure — NOT caused by this PR (tasks/pr307-ci-failure.md) + +- `gh pr checks 307`: server FAIL (run 40036094127, job 120695809498); CodeRabbit PASS; browser(1/2)+browser(2/2)+web pass. The one failing test: `not ok 366 - a rename that was cut off between its halves is finished only for the file it was decided for: a file with the two names that is another one is left as it is` at extensions/docs/test/rename.test.mts (stack line 759; subtest ~:740-771), error 'neither name of the other file was taken away', deepStrictEqual expected ['after.md','plan.md'] actual ['after.md']. PR touches nothing in extensions/docs; area introduced by main commit fc6d396c. +- Reproduced on a merge-base worktree WITHOUT PR code (1/5 full-file runs fail identically; branch 0/5; isolated subtest loops 0/30, 0/34 instrumented, 0/60 forced). +- Mechanism: the only path producing the observed disk state is the `#carryRename` recovery branch at extensions/docs/store.js:1108-1109 — it unlinks 'plan.md' via removeDocument when `recovering && old.isFile && now.isFile && same ino && nlink>1 && sameId(old,file)`, then commits a bogus rename revision. ROOT CAUSE (🟡 leaning 🔴, data-loss class): file identity decided from {dev,ino} ALONE — idOf at store.js:88 captures only dev+ino (captured at :1007/:1014); sameId :91 compares those two; the window spans the whole crash gap plus arbitrary other-actor activity; ext4 reuses inodes aggressively (~99% immediate reuse in create→unlink→create microbench). The test's own recreation (rename.test.mts:752-755) lands on the original inode when the allocator cooperates. workspace-files.ts `linkOver` already defends against exactly this with same-call re-stat+nlink; #carryRename has no equivalent. Second sameId use store.js:1164 (#carryDelete) is safe-direction — no change. +- Proposed fix (separate issue, not this PR): widen fingerprint to {dev,ino,size,mtimeNs} and require all four at :1108; optionally blobId(readBytes(from))===row.blob content check; harden the test with stub files that share dev+ino but differ in content. +- ACTION: rerun CI on PR #307 (unrelated flake); file a separate issue against extensions/docs. + +## CodeRabbit — latest unresolved comment (tasks/pr307-cr-latest.md) + +All other CR threads are resolved per the author. Thread id 4364892451 (2026-10-23T22:56:14Z), extensions/office/src/store.ts:384, 🩺 Stability & Availability / 🟠 Major / "🏗️ Heavy lift": "Keep import-time DOCX parsing off the portal event loop… Move parsing to a worker or enforce a suitable import-size limit before the synchronous read." + embedded ast-grep note '384-384: Filesystem path is not a string literal; … [CWE-22]'. + +- VERDICT: factually correct. server.ts:366 POST /import takes {path} from the JSON body, path.resolve, gates the parent with portal().folders.allowed(parent) (server.ts:372), then calls store.importFile synchronously (store.ts:362; statSync :371; parseDocx(fs.readFileSync(absPath)) for the docx baseline at :384; copyFileSync at :407). NO size cap anywhere; no HTTP upload (web/api.ts:77 sends only a path string); store header says the portal's 1 MB managed-file cap does not apply to the library by design. The office half runs IN the portal process (no "isolated" flag in package.json; core/server/src/extension-host.ts:887), so a blocking import starves all routes. +- Context: sync fs is established house style (load-bearing for index atomicity; ~30 core files readFileSync in request paths; same file does whole-file readFileSync on GET /documents/:id server.ts:340; office_read tool tools.ts:190 parses inline with no limit). ooxml caps exist pre-inflate: MAX_ENTRY_BYTES=256 MiB, MAX_TOTAL_BYTES=1 GiB (extensions/office/src/ooxml.ts:90,96). +- The CWE-22 note = our security finding N1 (folders.allowed checked on the resolved-but-not-realpath'd path; a symlink in an allowed folder points the import outside). Rated low impact (all principals share the host FS as the portal user); realpathSync + re-check (~4 lines) closes both notes at once. +- RECOMMENDED MINIMAL FIX (worker_threads NOT justified — it would break the deliberately-sync store contract / index atomicity and require a second parse pipeline for the equally-sync tool/persist paths tools.ts:190 and ping persist): + (a) run the docx baseline parse only up to a **32 MiB cap**; above it leave importedBaseline undefined → reuses the documented "unparseable source" fallback at extensions/office/src/ping.ts:235-240, preserving first-save baseline semantics; + (b) hard-reject imports above **512 MiB** with code:"too_large" (guards the sync copy + the 1 GiB envelope); + (c) fs.realpathSync(absPath) before the folders.allowed check. +- A ready-to-post response paragraph for the author is at the bottom of tasks/pr307-cr-latest.md. + +## PR's own checks (tasks/pr307-tools-verification.md, part B — ran in /root/repos/pithagoras) + +- `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` → 123 total / 120 pass / 3 skip (docker-gated; see below) / 0 fail, run without the flag. The PR body's claimed "120/117/3" is STALE (+3 passing since). + - Those 3 skips are NOT waiting on an OnlyOffice image: they gate on `OFFICE_IT_DOCKER=1` + a healthy `docker info` (plan M0 operator intent) and need no DS image or container, because the assertions run in-process against a synthetic DS stand-in. Locally verified 3/3 pass with the flag; now also enabled in CI by commit f6788220 (`OFFICE_IT_DOCKER=1` in test.yml's server job). +- `npx tsc --noEmit -p extensions/office` strict-clean (no root tsconfig exists; the PR's `-p .` only works from the extension dir). +- `node scripts/build-extensions.mjs` emits gitignored web/ + server.js — zero tracked pollution. + +## Open questions / follow-up candidates + +1. ooxml Q1: import-reject vs accept-warn for CRC-corrupt archives. +2. ooxml Q2 / cr-latest Q2: same size guard for GET /documents/:id (server.ts:340) and office_read (tools.ts:190) — same hazard, different routes; candidate follow-up ticket. +3. cr-latest Q1: one shared 512 MiB cap vs per-format (argue for one shared cap; copy is sync either way). +4. Mention grants permanent with no revocation path (ping.ts:168) — "for this task" promise unkept. + +## Merge notes / conflicts + +No findings conflict across the six review waves. Consistency points worth stating in the PR reply: the callback-concurrency file already accepted the sync-parse event-loop hazard class as scale-appropriate with a queue escape valve (matches the CR-latest minimal-fix stance), and CR's ast-grep CWE-22 note is our own security finding N1, answered by the same realpathSync line. + +## Findings files + +- tasks/pr307-triage-architecture.md +- tasks/pr307-security.md +- tasks/pr307-ooxml.md +- tasks/pr307-callback-concurrency.md +- tasks/pr307-tools-verification.md (tools review + PR self-checks) +- tasks/pr307-ci-failure.md (unrelated docs bug, repro + fix) +- tasks/pr307-cr-latest.md (latest CR thread verdict + ready-to-post reply) +- tasks/pr307-coderabbit-comments.md (CR top-level comments archive) + +## Environment / cleanup + +Repo left clean: no worktrees beyond the main checkout, no orphaned test processes, git status shows only pre-existing untracked files (tasks/, PLAN-office-280.md, .office-contracts.md, .pi/, .reports/). Ephemeral /tmp/pithagoras-* test dirs from runs were left in place. No fixes applied to PR code; no scratch files created by the review itself. + + +## Fixes implemented (pushed) — commit f4330519 "Office extension: address code review findings (#307)" + +22 files, +606/−109, fast-forward push fee666ce..f4330519 to thecodacus/pithagoras branch office/280-pithagoras-office; PR #307 tip confirmed at f4330519. Independent re-verification: 135 pass / 3 docker-gated skip / 0 fail, tsc strict clean, build OK, built server has no static jszip import. + +**Core (tasks/pr307-fixes-core.md):** +1. 🔴 Dense-grid cap in ooxml.ts parseXlsx — MAX_SHEET_CELLS = 256*1024 per sheet + MAX_TOTAL_GRID_CELLS = 1024*1024 archive budget; height shrunk before allocation, skips surfaced via existing truncatedSheets. Attack shape (3 sheets × sparse row-20000/col-500 cells) now parses in ms with honest truncation. +2. Import guards — server.ts realpathSync before the folders.allowed gate (CWE-22 symlink escape closed, tested); store.ts 512 MiB hard cap → too_large/HTTP 413; docx importedBaseline only ≤ 32 MiB else undefined (ping first-save fallback applies). +3. CRC-32 validation on zip read — structured invalid_zip "CRC mismatch" throw in readEntryData; corrupt archives fail loudly instead of being re-persisted (byte-flip test added). Resolves open question Q1 as import-reject/fail-loud. +4. office_comments output cap — mirrors office_read: shared 200,000-char limit + truncation marker, details bounded 100 × 500 chars with true total in commentCount. +5. Presence TTL in live.ts — 15-min default (matches config-signature lifetime), setPresenceTtl for tests, heartbeat refresh on every setUser(true) so live sessions never trip it; stale entries drop lazily on read. Co-edit refusal semantics unchanged. +6. Token purpose separation — signed typ claim in portal-minted JWTs (editor-config / fetch) + optional purpose pin in verifyJwt; tickets HMAC over "document-ticket|docId|expMs". No key change → adopted DS containers unaffected. (Deliberately NOT done: per-purpose key derivation, because DS verifies the config signature against the raw env secret.) +7. rewriteZip case-insensitive dedupe — case-variant part names now overwrite in place (original name kept); no more shadow parts. + +**Docs/wiring (tasks/pr307-fixes-docs.md):** +1. Co-edit docs corrected to REAL semantics (behavior confirmed intentional by author): README.md:11, src/web/settings.tsx:109 tooltip + three spots in src/tools.ts now state ON ⇒ agent edits refused (error:session_open) while a human session is open; OFF ⇒ proceeds under the mtime gate. Logic/tests untouched. +2. deps.js wired (docs pattern): new src/jszip.ts lazy loader (jsZip()/armJsZipInstall), store.ts+tools.ts load lazily, server.ts boots void armJsZipInstall(ensureDependencies(logLine)), build.mjs uses deps.js missingDependencies, new deps.d.ts. Verified: built server imports fine with node_modules/jszip renamed away; missingDependencies() reports ["jszip"] when absent. +3. Export wording fixed — README says "import into the library and export (download) a copy"; "Exporting into chat folders" added to Deferred. +4. Single index writer — new exported store.addGrant (readIndex → dedupe → atomic writeIndex); ping.ts's hand-rolled read-modify-write deleted, thin wrapper keeps its vanished-doc log line. +5. Wake wording fixed — "…for this task" → "You have been given edit access to this document"; ping.test.mts template assertions updated; expiry remains deferred. +6. README "Security notes" section — host netns ⇒ DS nginx on 0.0.0.0:80 visible to host/LAN/tailnet; firewall :80 or (planned) named-volume nginx override. (Container setup deliberately unchanged — no OnlyOffice image available locally to test.) + +**Deliberately not done:** 🟢 cross-paragraph quote-edit stray line break (needs paragraph-level XML surgery, documented in pr307-fixes-docs.md); host-mode nginx conf bind change (untestable without DS image). + +**Remaining follow-ups:** size gate for GET /documents/:id + office_read on huge workbooks (wall-clock guard); mention-grant revocation/expiry; per-purpose key rotation. CI server job will re-run — the prior failure is the unrelated extensions/docs {dev,ino} flake (separate issue to be filed against that extension). + +## Workspace library + co-editor push + +Verified independently on top of pushed tip f4330519, then committed and pushed as one commit. + +- **Tests** — `env -u NODE_ENV node --import tsx --test "extensions/office/test/*.test.mts"` run 6 times in a row: + every run **143 tests / 140 pass / 0 fail / 3 skipped** (the 3 pre-existing docker-gated integration skips; OFFICE_IT_DOCKER unset), all exit 0. +- **tsc** — `npx tsc --noEmit -p extensions/office` → exit 0, no errors. +- **Build** — `node scripts/build-extensions.mjs` → exit 0 (web/ + server.js regenerated); `git status` clean of tracked pollution afterwards (build outputs are gitignored). +- **Coherence spot-checks** — settings shape is `{enabled, imageTag}` only in web/api.ts types, the POST /settings 400 path ("send { enabled?, imageTag? }") and GET /state payload; zero `coEdit` identifiers remain outside legacy-tolerance comment + tests; zero `session_open`; office_edit stale paths answer `error:revision_mismatch` with a loud CONFLICT naming open editors; POST /import route and ImportDocuments UI are gone; scanWorkspace runs at boot after settleKept and on POST /documents/rescan; README describes the no-switch co-editor model + automatic scan. +- **Scope** — changed set confined to `extensions/office/**` + doc artifacts (`tasks/`, `PLAN-office-280.md`). Untracked `.office-contracts.md` (its own header says "local scratch, never commit"), `.pi/` and `.reports/` were deliberately left uncommitted. +- **Commit** — `e4fd0b51716f7e589443584397090e2be6d28756` "Office extension: agents co-edit alongside humans; workspace scan instead of manual import" (no trailers, matching the PR's prior commit style), 32 files (+3045/−375): extensions/office {README.md, style.css, src/{live,ping,server,settings,store,tools}.ts, src/web/{api.ts,library.tsx,settings.tsx}, test/{container,integration,server,settings,store,tools}.test.mts} + PLAN-office-280.md + tasks/ report set. +- **Push** — fast-forward f4330519..e4fd0b51 to origin; `gh pr view 307 --json headRefOid` == e4fd0b51… ✓. At the moment of the push no CI checks were reported on the branch yet ("no checks reported" — created later, not waited on). + +### DEPLOYMENT NOTE (read before assuming anything is live) + +The live portal process is **still running a stale pre-rebuild extension bundle** (it started before the noon rebuild and never reloaded; its data dir had no `index.json`, and it accepted a `coEdit` setting post-change — only the old code has that key). **None of these behaviors (no coEdit switch, CONFLICT model, workspace scan/rescan, sortables, 503 dependencies_pending) are visible to the user until the portal restarts/deploys.** The verify agent did NOT restart or kill any process. + +## Proxy encoding bug (user m00343: editor script at /api/x/pithagoras-office/ds/... could not be loaded, after reload with bfb5edc5 live) +- Live diagnosis: portal port 4100, routes behind session gate (unauth 401); SSO-only login (password switched off), forged a valid pi_portal_auth cookie locally (sha256(PORTAL_SECRET+\0+identity) HMAC) and probed with it: /state 200, both /ds/ URLs => connection 'terminated' mid-response. +- ROOT CAUSE: Node undici fetch AUTO-DECODES gzip but keeps upstream content-encoding: gzip + COMPRESSED content-length (live api.js: CL 13720 vs 67458 decoded bytes). Proxy copied both, res.end(67458B) => Node emitted Content-Length: 13720 over a 67458-byte body => truncated invalid api.js + corrupted keep-alive stream in every remote browser. +- FIX cefef592 (pushed; PR #307 tip confirmed): /ds fetch now sends accept-encoding: identity (OnlyOffice nginx honors it); content-encoding + content-length no longer copied -- Node sizes the answer from the actual buffer. New regression test: hostile fake DS answers the identity ask with gzip body + lying headers, asserts full original payload arrives intact. Findings: tasks/pr307-ds-proxy-fix2.md (+ pr307-ds-proxy.md). +- Test wrinkle hit along the way: first draft of that regression test asserted gzip(85-byte API_JS_BYTES stub) < original -- false for tiny payloads, and its throw skipped boot() cleanup so the whole node --test process never exited (looked like a hang). Rewrote with 64x repeated payload + always-run finally cleanup. +- Finish line: 151 total / 148 pass / 0 fail / 3 docker-gated skips; tsc -p extensions/office exit 0; build exit 0. User reloaded once more to pick up cefef592; editor load pending their confirmation. + +## Editor stuck on "Loading the editor…" (user m00574) — DocEditor placeholder bug, fixed 2026-07 (commit pending tip) +- Symptom: after cefef592 was live and the proxy served api.js byte-perfect (live-verified: 67458 B, CL matches), the page still sat on the loading state forever; no error banner, editor never appeared. Deterministic, reproduced in a logged-in browser. +- ROOT CAUSE: extensions/office/src/web/office-editor.tsx passed the host ELEMENT to `new window.DocsAPI.DocEditor(box.current, config)`, but api.js declares `DocsAPI.DocEditor = function(placeholderId, config)` and does `document.getElementById(placeholderId)` then `target.parentNode.replaceChild(iframe, target)`. An element stringifies to "[object HTMLDivElement]" => getElementById returns null => the constructor silently builds nothing (no iframe, no alert, no error) => the 500 ms ready-poll never fires. +- FIX: the effect now appends its own `div#po-doceditor` placeholder INSIDE the host box but OUTSIDE React's children, and passes the id string; cleanup sweeps `document.getElementById("po-doceditor")?.remove()` (no-op after a successful build, since DS replaces it with the iframe; remount safety removes any stacked leftover first). Type updated: `DocEditor: new (placeholderId: string, ...)`. +- Live verification (browser logged in on 127.0.0.1:4100, running portal picked up the rebuilt web/ without restart): sample.docx renders the word editor, sample.xlsx the spreadsheet editor; "N editing now" badge intact. The suspected secondary blocker (sdkjs/vendor/string.js 404 under the proxy prefix) did not prevent boot. +- Finish line: office suite 151 total / 148 pass / 0 fail / 3 docker-gated skips; tsc -p extensions/office exit 0; build exit 0. + +## Document loads but never opens (white spinner) — the /ds proxy was GET-only, fixed in 13308687 +- Symptom: after e818ddc the editor iframe booted (api.js + app assets load) but the page sat on DS's white loading animation endlessly; docserver never logged document activity for our keys. +- Protocol replay through the proxy (logged-in fetch): `GET …/doc/<key>/c/?EIO=4&transport=polling` answers a 120-byte engine.io v4 handshake (`0{"sid":…,"upgrades":["websocket"],"pingInterval":25000,…}`); the browser's stuck loop = fast ~302 B handshakes + upstream 499 at exactly 15.000 s, repeating ~1/s (DS nginx log), zero successful document sessions. +- ROOT CAUSE (proven): `/c/` is the editor app's socket.io POLLING channel — after the handshake its packets arrive as **POSTs** (`"40"` connect packet etc.). Our /ds route was registered GET-only, so every POST died at the portal with the core's 404 "No such API route" (core/server/src/server.ts:1723); the channel could never open. Second wrinkle in the same handler: `AbortSignal.timeout(15_000)` killed healthy idle long-polls — undici abort closes the upstream socket, which docserver records as the 499 at precisely that moment (engine.io pings every 25 s, so polls routinely sit empty > 15 s). +- FIX 13308687 (pushed; PR #307 headRefOid confirmed == 13308687): handler now serves GET **and** POST — non-GET/HEAD bodies are streamed through verbatim (128 MB cap, above the handshake's maxPayload 100 MB) with the client's content-type forwarded; the abort window grew to 90 s (a full ping cycle + slack); and outbound handshakes get `websocket` stripped from `upgrades` (the fetch-based proxy cannot relay WS frames — pinning clients to polling removes failed-upgrade churn), re-serialized behind the preserved `0` EIO packet prefix. +- Tests: three new /ds cases in test/server.test.mts against a fake engine.io endpoint — POST forwards method + raw body + query verbatim (byte-for-byte "40"), handshake stripping keeps sid and drops websocket, and an idle poll HELD 16.5 s still answers 200 (elapsed >= 15 s asserted) guarding the old abort class. MiniRouter gained a streamable fake request (asyncIterator over body + headers arg) to exercise the POST path. +- Finish line: 3 consecutive full-suite runs 154 total / 151 pass / 0 fail / 3 docker-gated skips; tsc -p extensions/office exit 0; build exit 0. Push was a single-commit fast-forward (cefef592 + 51bc708d were already in origin via e818ddc's ancestry). +- DEPLOYMENT NOTE: server.ts changed — the running portal process must be reloaded/restarted for the fix to be live (a browser-only reload is not enough; look for a fresh "[extension pithagoras-office] server half started" journal line), then reopen the document.