feat: Microsoft 365 (Graph) human-selected document ingestion connector (#492) - #514
Open
mavaali wants to merge 52 commits into
Open
feat: Microsoft 365 (Graph) human-selected document ingestion connector (#492)#514mavaali wants to merge 52 commits into
mavaali wants to merge 52 commits into
Conversation
…osoft (U1) Introduces PROVIDER_NAMES as the single source of truth for the closed ProviderName union and derives every union/regex/iteration edit site from it (types.ts, state.ts, runtime.ts, routes.ts, queue.ts, utils/config.ts). Adds "microsoft" as a first-class provider with a stub adapter factory in runtime.ts that throws until the real Microsoft adapter lands. Google and Notion behavior is unchanged; full existing suite plus new isProviderName and providerFrom regression tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dators (U2) Add EnrollmentRecord, ProviderState.enrollments/account/authorization, SourceState.enrollmentId/lastFailure (SourceFailureReason), WebhookChannel .subscriptions, ProviderTokens.account, a VerifiedWebhook "lifecycle" variant, WebhookRequest.query, and UnavailableSourceEvent's "unenrolled" reason. All fields are optional and validated in state.ts so old encrypted envelopes still parse and new ones round-trip losslessly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add rejection tests for an unknown SourceFailureReason, a malformed EnrollmentRecord (missing field and wrong kind), and a ProviderAccount missing tenantId, exercised through the real writeIntegrationState path. Also give validSourceFailure/validProviderAccount/validAuthorization type-predicate return types to match the file's other validators. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Threads an optional `collection` field through DistillationInput -> DistillUpsertInput -> DistillIds -> proposeAllClaims, so a distill proposal can target an allowlisted collection instead of the hardcoded `distill`. Unset or empty-string collection falls back to DISTILL_COLLECTION exactly as before, keeping the default path byte-identical. No allowlist/validation added here (deferred to a later route task). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(U3 follow-up) Comment-only follow-up from U3 quality review. Clarifies that the new `collection` parameter in derivePath is NOT covered by the slugifyKey traversal-safety invariant (unlike sourceId/title), and that U19 must add allowlist validation before any provider with untrusted collection input uses this path. Also repoints the stale file-header defaulting comment at the actual default-resolution site in proposeAllClaims. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(U6) Provider-neutral extractText(bytes, kind, limits) runs document extraction in a bounded worker_threads Worker: a wall-clock timeout terminates the worker on overrun (-> timeout), heap is capped via resourceLimits. The dispatch table in worker.ts is the only edit U7/U8/U9 need to plug in real docx/pptx/pdf extractors; each is currently an unsupported_type stub. extractText accepts an injectable workerUrl/execArgv seam so tests can supply a slow or fixed-result worker without touching the real dispatch table or spawning real parsing logic. normalize.ts implements the deterministic CRLF/trailing-whitespace/blank-run rules from spec section 3.2 and mirrors readtext.ts's NUL-byte guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d (U6 follow-up) A worker terminated for exceeding resourceLimits.maxOldGenerationSizeMb was being reported identically to a generic parser crash (both "malformed"), so a caller acting on lastFailure.reason couldn't tell "give up, unsupported" from "too big, could retry with lower limits". Node signals the heap-cap kill via a worker "error" event carrying code ERR_WORKER_OUT_OF_MEMORY (confirmed empirically: error fires before exit) — detect that code and map it to the existing too_large taxonomy entry; a generic thrown/uncaught worker error still maps to malformed. Added a fixture worker (oom-worker.ts) that allocates past a small workerHeapMb and a harness test asserting the too_large mapping — verified non-flaky over 5 consecutive runs, ~70ms each. Also documented that workerHeapMb (maxOldGenerationSizeMb) isn't a hard process-memory ceiling: the worker gets maxYoungGenerationSizeMb on top, so worst-case heap is ~1.25x the configured value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds src/extract/office.ts: a linear, non-DTD/entity-expanding OOXML tokenizer + tree builder, a declared-size-capped zip opener (fflate unzipSync filter, zip-bomb defense on DECLARED sizes before inflation), and a WordprocessingML driver that walks word/document.xml (+ footnotes/endnotes), preserving paragraphs/tables/tabs/breaks/text-boxes, keeping w:ins and dropping w:del, decoding standard XML entities, and normalizing per spec §3.2. Wires the docx entry in worker.ts's dispatch table (pptx/pdf stubs untouched). Also fixes a latent dev/test-only bug this unit's real worker dispatch exposed: tsx's ESM resolve hook doesn't reliably remap a ".js" specifier to its sibling ".ts" source for value imports resolved inside a worker_threads realm, so worker.ts and office.ts now resolve their one hop of sibling imports via an exact-extension URL + dynamic import instead of a plain specifier (production/compiled dist is unaffected — plain Node ESM resolution there, no loader involved). Adds fflate as a runtime dependency (approved for this task). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…U7 follow-up) - readOoxmlParts's `wanted` param now accepts a predicate (readonly string[] | ((name: string) => boolean)) alongside the existing exact-name array, so U8's pptx driver can select ppt/slides/slideN.xml for an a-priori-unknown N without re-implementing the zip opener. Docx's existing array-form call is unchanged (same behavior). Adds tests for both the predicate and array forms. - Doc-only guards for future readers/reusers: recursive tree-walk relies on V8's catchable RangeError (not a bounded-depth guard) -> malformed via worker.ts's top-level catch; decodeUtf8Lenient is intentionally lenient since normalize()'s assertUtf8NoNul/containsNulByte still enforces the NUL check downstream; table-cell/footnote joins intentionally collapse multi-paragraph content to one line given the one-line-per-row/note output format. No behavior change for docx. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ip (U8) Adds the .pptx (DrawingML/PresentationML) driver alongside U7's docx driver, reusing every shared primitive (readOoxmlParts, tokenizeXml/parseXml, decodeXmlEntities, getAttr, the OLE encrypted check, normalize()) with no duplicated zip/tokenizer logic. Wires the pptx worker-dispatch entry; pdf stays a U9 stub. Slides are emitted in presentation order (resolved via p:sldIdLst's r:id through presentation.xml.rels), not slideN.xml filename order. Hidden slides (show="0") are skipped entirely and excluded from the visible 1-based "## Slide N" numbering. Tables (a:tbl) and grouped shapes (p:grpSp) are walked recursively; speaker notes are resolved per-slide via its _rels part and appended under a "[speaker notes]" line. Threads a new includeSpeakerNotes flag (default true) from extractText's signature into ExtractRequest, so it structured-clones into the worker alongside limits — kept off the existing ExtractOptions bag since that's a main-thread-only Worker-spawning seam, never sent to the worker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…w-up) Adds 3 tests exercising graceful degradation on the pptx relationship- resolution surface (the newest untrusted-input path from U8): a dangling r:id in presentation.xml.rels drops just the unresolved slide, a wholly missing presentation.xml.rels drops all slides (-> empty), and a dangling notes-relationship target keeps the slide body but omits the [speaker notes] block — none throw. Also adds a one-line recursion-depth caveat on the pptx shape-tree walkers (collectRunTextPptx/collectParagraphsPptx/collectShapeTreeLines/ findDescendant), mirroring the existing docx note on collectRunText: no explicit depth cap, relies on V8's catchable RangeError -> malformed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the pdf entry of the worker dispatch to a real extractor: pdfjs-dist's legacy Node build parses the text layer, with strict encrypted/empty/ too_large/malformed classification per design spec §3.2/§3.3. Mirrors office.ts's exact-extension sibling-import pattern so pdf.ts resolves correctly from inside a worker_threads realm under tsx in dev/test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e floor (U9 follow-up) Move standard-fonts directory resolution out of pdf.ts's module top level and into a lazy, memoized getter called from extractPdf(). worker.ts imports pdf.ts unconditionally before building its dispatch table, so a top-level throw there (e.g. a packaging layout stripping pdfjs-dist's standard_fonts/) would previously take down docx/pptx extraction too. On resolution failure, extraction now degrades (extracts without standardFontDataUrl) instead of failing outright. Also corrects engines.node to >=20.9.0: import.meta.resolve is synchronous and unflagged only from Node 20.6+, so the prior >=20 floor overstated actual compatibility. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-loss evidence (U10) Consolidates docx/pptx/pdf extraction evaluation into a single data-driven corpus (test/extract/corpus.test.ts): 100% decision-sentence recall on happy fixtures, exact failure-reason classification on negatives (encrypted/ malformed/empty/too_large per type), and a measured text_chars/source_bytes ratio per type written to test/fixtures/extract/ratios.json for U17's enrollment cost-preview to read. Also records extraction-level evidence that routing a deck through PDF drops speaker notes, supporting design override #1 (keep Graph ?format=pdf out of the primary Word/PowerPoint path). Factors the existing per-format fixture builders (office-docx/pptx/pdf.test.ts) into test/extract/fixtures.ts so the corpus can reuse them without duplicating OOXML/PDF construction and without exporting from *.test.ts files (disallowed by lint/suspicious/noExportsInTest) — no extractor behavior changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…w-up) The ratios test previously recomputed the per-type text_chars/source_bytes ratios and unconditionally overwrote the committed ratios.json before reading it back — a tautology that could never catch drift in extractor output, leaving U17's cost-preview trusting a number with no real gate behind it. Now the test reads the committed ratios.json first, recomputes the ratios from the happy fixtures, and asserts each measured value matches the committed value within a tight tolerance (toBeCloseTo, 6 digits) — it never writes to disk during a normal run. Regeneration is opt-in only, via UPDATE_RATIOS=1. Also drops the no-op PDF_TOO_LARGE_LIMITS override (maxPdfPages: 500, same as the default) in favor of a genuinely smaller cap (10) with a fixture of 11 pages, so the too_large branch is exercised independent of whatever DEFAULT_EXTRACT_LIMITS.maxPdfPages happens to be. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tate + carry enrollments on reconnect (U4) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es, not transient (U4 follow-up) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ches + pin terminal-message golden (U4 follow-up) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cycle routes (U5) Adds the provider-neutral OPTIONAL adapter-method surface (answerWebhookChallenge, verifyLifecycleWebhook, resolveEnrollment, estimateEnrollment, describeStatus) plus their supporting types (EnrollmentContext, EnrollmentDraft, EnrollmentEstimate, ProviderStatus) that the Microsoft connector will implement in later units. Wires the webhook-side routes: the /webhook route now answers a provider's validation challenge directly (ahead of signature verification, no state/queue touch), and a new /webhook/lifecycle route verifies and durably enqueues a lifecycle notification, replacing the U2 compile shim that only ack'd without enqueueing. Both routes 404 for a provider lacking the corresponding optional method. Google/Notion are unaffected — they simply don't implement these methods. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pse to reconcile (U5 follow-up)
The `action` (reauthorize/recreate/reconcile) on a verified lifecycle webhook
existed only at the enqueue site; hardcoding hint:{kind:"reconcile"} there
destroyed it before U16 (which executes reauthorize/recreate in-cycle) could
ever read it back, and RefreshHint had no variant to carry it even if it had
survived.
Adds a `{ kind: "lifecycle"; action }` variant to RefreshHint, threads it
through queue.ts's validHint/validQueueItem, and makes mergeHints treat a
lifecycle action as never fungible with a plain reconcile: any lifecycle item
in a drain batch wins the merge over an unrelated reconcile item, and when
multiple lifecycle items coalesce the strongest action (reauthorize >
recreate > reconcile) survives. Both webhook-route enqueue sites (the
/webhook lifecycle branch and the dedicated /webhook/lifecycle route) now
pass the verified action through unchanged instead of a hardcoded reconcile.
reconcileProvider itself still has no lifecycle-action dispatch (that's
U16's job) — a lifecycle hint reaching it directly falls back to the same
full-discovery path a reconcile hint gets, which is safe and lossless but
adds no new execution semantics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s + distill usd key (U11) Adds MicrosoftProviderConfig (tenant_id, scope_profile, collections, include_speaker_notes, picker_host) validated against its own recognised-key table, replacing the single flat RECOGNISED_INTEGRATION_PROVIDER_KEYS with a per-provider table so Google/Notion cannot accept Microsoft-only keys and vice versa. Also adds the optional distill.estimated_usd_per_call config key (R39) — absent leaves downstream USD estimation disabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (U11 follow-up) RECOGNISED_MICROSOFT_PROVIDER_KEYS is hand-maintained; nothing previously forced it to stay in sync with MicrosoftProviderConfig. Adds a compile-time Record<keyof MicrosoftProviderConfig, ...> mapping that fails to typecheck if a config field is added without a matching snake_case key, plus a runtime test that populates every recognised Microsoft key at once and asserts a clean parse with no unknown-key error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…est harness (U12) Adds createMicrosoftAdapter as a capability-VALID ProviderAdapter (Graph/Entra HTTP stays confined to this file) and wires it into the runtime factory in place of the U1 throwing stub. Real behavior lands in U13-U18; this unit is scaffolding plus the R40 injected-transport fixture harness later Microsoft adapter tests will import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ess scope notes (U12 follow-up) Gives fetchSource its typed (source, state) params instead of a zero-parameter signature that only compiles by structural luck, swaps its inline return-type duplicate for NormalizedRemoteSource, and documents why it throws instead of returning err(...) like exchangeCode does. Also notes that createFixtureTransport only scripts canned responses and discards `init`, so U13/U16 need a bespoke capturing transport for request assertions. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h (U13) Implements the three real OAuth methods on the Microsoft adapter, replacing the U12 placeholders: authorizationUrl (scopeProfile -> Files.Read / Files.Read.All + offline_access/User.Read/openid, prompt=select_account), exchangeCode (token POST, id_token tid decode with an /organization fallback, /me lookup for ProviderAccount), and refreshTokens (rotated refresh token, terminal-signal .status/.terminal on the response so engine.ts's isTerminalRefreshError can classify 400/401/403 precisely without message sniffing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…json helpers across adapters (U13 follow-up) Addresses code review of 667b939: 1. Microsoft's requestJson now goes through the same bounded (8MiB) + timed-out (30s) transport/JSON-parse path Google already had, instead of a bare transport(...).then(r => r.json()) that could hang or buffer an unbounded body. A timeout or size-cap trip returns a clean, non-terminal err() (no .status/.terminal), so the engine retries rather than reconnect-prompting. 2. Extracted the shared, byte-for-byte-identical stringValue/tokenExpiration plus the bounded+timeout request machinery (providerResponse/boundedJson/ jsonResponse) into src/integrations/http-json.ts, parameterized by a providerLabel so each adapter keeps its exact error message text (google's golden-pinned "Google request failed with status 400" is unchanged). Refactored google.ts onto the shared helpers (behavior-preserving — full google/notion/engine suites pass unchanged) and moved TERMINAL_REFRESH_STATUSES out of engine.ts into the shared module so both engine.ts's classifier and microsoft.ts's error-tagging import one set. 3. Added a malformed-id_token test (falls back to /organization without throwing) plus timeout/size-cap tests for the new bounded requestJson. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tract (U14) Replace the U12 discover placeholder with real Microsoft Graph delta discovery (R20, R21, R22, R23, R37, §8): container enrollments walk a folder-scoped delta (with a probe-deferred drive-root fallback on 400 plus manual ancestry tracking since delta omits parentReference.path); item enrollments group per drive and walk root/delta?token=latest so a large library is never fully enumerated. Cursor threading mirrors google.ts exactly (discover mutates state.cursor in place, only once every root has succeeded), 410 triggers a per-root resync without disturbing other roots, 429 honors a single bounded Retry-After retry, and pagination is bounded with a repeated-link guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rrect remembered-set comment (U14 follow-up)
Add coverage for the previously-untested container 400->drive-root-delta
fallback (the most fragile part of container discovery: ancestry is seeded
with only {folderId} while walking a delta feed that can return
out-of-subtree items before any subfolder hierarchy is known). Also correct
the remembered-set/driveId-prefix comment: the real risk is availability-state
corruption (a deleted item in one root can be silently resurrected by another
root sharing the same driveId, defeating R37), and the missing fix is
downstream engine plumbing (RemoteSource/SourceState carry no root tag,
sourceState() never sets enrollmentId) rather than widening EnrollmentRecord,
which already carries driveId/cursorKey/id. A follow-up bead tracks the
engine-contract fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…wo-pass classify to stop dropping nested files (U14 fix) Critical fix (quality review of dd0d329): the container ancestry set was rebuilt from scratch (new Set([folderId])) on every discover() call, but a resumed incremental delta only sends CHANGED items — Graph never resends an unchanged ancestor folder record. A file 2+ levels under the enrolled folder, edited while its parent subfolder is unchanged, arrived with a parentReference the walk had never seen, so it was misclassified "outside subtree" and silently dropped. Silent data loss on ordinary nested-folder edits. Two-part fix: 1. The PRIMARY container path (folder-scoped delta) no longer does any ancestry/parent reasoning at all — Graph itself scopes every item that endpoint returns to the requested folder's subtree, so trusting that removes the bug entirely for the common case. 2. The drive-root FALLBACK path (used only when folder-scoped delta 400s) is the one place ancestry filtering is genuinely needed, since it walks the whole drive. It now: persists discovered subtree folder ids in the cursor (extending a root's cursor entry from a bare deltaLink string to {link, folders[]}, parsed defensively/backward-compatibly) and reseeds ancestry from that set on every resumed cycle; runs a two-pass fixpoint fold over every item collected across the WHOLE walk before classifying anything, so intra-page/intra-walk ordering can't hide a subfolder's record arriving after its child's; and never removes a previously-tracked item on an unresolved/ambiguous parent — only an explicit `deleted` facet (or a full resync) removes, trading a possible late remove for never silently losing an in-scope item. Adds regression coverage for the exact bug (resumed primary walk, nested file whose parent record is absent from the page), two-pass ordering-independence in the fallback walk, last-occurrence-wins across a page boundary, a deleted item reappearing later in the same stream, and a fallback-mode root's 410 resync skipping straight to the drive-root endpoint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…imary-scoping + multi-level nesting coverage (U14 final) Doc/test-only follow-up (re-review confirmed the ancestry fix itself is sound). Four items: 1. applyContainerFallbackItem's conservative-on-ambiguity comment now states plainly that it DEVIATES from R21 (out-of-subtree removes) and R37 (move-out -> available:false) in the fallback path only: a genuinely moved-out item lingers as available with a stale revision until an explicit deleted facet or a full resync, rather than resolving to available:false immediately. The primary folder-scoped path is unaffected (Graph simply stops returning a moved-out file, so the engine's ordinary not-returned -> unavailable rule still fires). A filed bead tracks revisiting this once probe 2 confirms folder-scoped delta's real-world 400 rate. 2. Added a fallback-mode test proving the moved-out-lingers behavior is tested, not accidental: a previously-tracked item's parent moves outside the known subtree and the item remains present with its prior revision. 3. classifyContainerPrimary's comment now flags "folder-scoped delta is subtree-scoped" as the same probe-2-dependent assumption as the 400 trigger (containerFallbackUrl already flagged that one) — if wrong, the primary path would under-filter (include out-of-subtree items), a real but lower-severity residual risk than the Critical bug this unit fixed. 4. Added a 3+-level nesting test for the two-pass fixpoint (grandparent -> parent -> file, delivered file-first) to lock in multi-level resolution beyond the existing 2-level coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing + legacy pdf conversion (U15) Implements the U12 fetch placeholder: metadata precheck (malware/size/type) before any download, redirect-follow to Graph's pre-authenticated content URL WITHOUT the Authorization header (R21 SECURITY), extension-based routing to src/extract, legacy .doc/.ppt via ?format=pdf conversion (R28, labeled pdf-conversion), R30 short-circuit on a durable prior failure with an unchanged revision, and includeSpeakerNotes resolution from the owning enrollment with a config fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion routing (U15 follow-up) Closes two coverage gaps flagged in spec review of 6e1e4cd: the fetch path's requestWithRetry (metadata + content GETs) had no dedicated 429 test, and .docx/.pdf extension routing was only exercised indirectly via .pptx/.ppt. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ll hardening on microsoft fetch (U15 follow-up) Addresses quality-review hardening items on the fetch redirect/error-tagging branches: every failure branch in fetchSource now carries an explicit .reason (metadata non-2xx, missing-Location, content non-2xx all tagged "fetch", consistent with the malware/size/type/permission branches next to them); a second redirect from the SAS host is now detected and rejected explicitly (single-hop policy, documented, no unbounded follow); the malware facet check is tightened from `!== undefined` to a truthy/object check so a `malware: null` facet (a real Graph serialization some tenants use for "no malware") isn't false-positived as malware; and the SAS request's retry-asymmetry (no requestWithRetry — a blob-storage single-file GET, not a throttled Graph API call) is now explicitly commented. No behavior change beyond the malware-null fix and the newly-explicit double-redirect rejection (previously fell through to a generic, untagged status error — same failure outcome, now tagged and intentional). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y/lifecycle (U16) Implements ensureWebhook (one Graph subscription per enrolled drive, fanned out under a stable channel id/clientState secret; keep/PATCH/POST/DELETE per-drive lifecycle; R19 polling fallback on a non-HTTPS callback), answerWebhookChallenge (stateless validation-token echo), verifyWebhook (timing-safe clientState + known-subscriptionId verification, reconcile hint), and verifyLifecycleWebhook (lifecycle event -> queued action mapping), replacing the U12 throwing stubs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…per-drive dedup (U16 follow-up) Closes two coverage gaps flagged in spec review of 8af12c6 (no behavior change): a multi-entry verifyWebhook notification batch where only one entry is invalid (wrong clientState, or an unknown subscriptionId) now asserts the WHOLE batch is rejected; and ensureWebhook with two enrollments sharing the same driveId now asserts exactly one POST /subscriptions is issued for that drive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ompare in http-json (U16 follow-up) ensureWebhook now deletes any stored Microsoft subscription that isn't claimed by a currently-enrolled drive this cycle — including one whose `resource` doesn't parse to a driveId at all — instead of silently dropping it from tracking while the real Graph subscription lives on untracked against the §11 limit. Consolidates the timing-safe secret-compare (`equalWebhookSecret`/ `equalSecret`) and `validHttpsUrl`, each verbatim-duplicated three times across google.ts/microsoft.ts/engine.ts, into one canonical `timingSafeSecretEqual`/`validHttpsUrl` pair in http-json.ts. Behavior- preserving: google/notion/engine suites pass unchanged. Also guards `lifecycleNotificationUrl` against a double slash on a trailing-slash callbackUrl, and rewords the minted-event-id comment (a fresh random id means the queue's dedup does NOT catch a Graph retry — it's re-enqueued, which is harmless only because the reconcile hint is idempotent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…preview (U17) Implements resolveEnrollment (validates an untrusted File-Picker payload server-side, expands containers via U14's delta walk, enforces the 1,000-eligible-per-folder and 2,000-source-per-vault bounds) and estimateEnrollment (metadata-only cost/audience preview: byte->char ratios seeded from U10's ratios.json, LLM call estimate, readers/ratifiers, lossy-legacy-conversion warnings) on the Microsoft adapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… + cover 404/malformed/malware/dupes (U17 follow-up) The 2,000-source enrollment bound only ever sees this Microsoft adapter's own ProviderState.sources, not a cross-provider vault total — reword the comment and failure message to say so precisely, and note that a true cross-vault cap needs an engine/route-level check (bead tracks it). Also dedupe a picker payload referencing the same item twice, and add the review-flagged coverage: 404 (not just 403) rejection by name, non-array/ empty selection, a malformed selection entry, a malware-flagged item, and a duplicate item id resolving to a single draft item. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ate re-fetch + share item validation (U17 follow-up) resolveEnrollment now caches each item's size and each container's already- expanded eligible children on EnrollmentDraft.items (extended provider- neutrally in engine.ts), so estimateEnrollment reads that cached data instead of re-fetching item metadata / re-walking a container's folder delta for the common resolve-then-estimate preview flow — eliminating a full second Graph round-trip per item/container. A caller that POSTs a bare draft straight to /estimate without cached data still works via a stateless fallback that re-fetches/re-walks exactly as before. Also extracts the duplicated isFile/malware/classifyExtension check from resolveEnrollment and estimateEnrollment's stateless fallback into a single validateEnrollmentItemMetadata helper, so the two paths can't silently drift. Notes the skipped-reason vocabulary is preview-only (not SourceFailureReason), and warns when RBAC roles aren't configured so an empty readers/ratifiers list isn't misread as "nobody can read this." Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ention (U18) Implements the optional describeStatus(state) on the Microsoft adapter, a pure projection of ProviderState into connection/webhook/enrollment/source status (design §13) plus the constant R34 sensitivity-labels disclosure. Also adds provider-neutral last-ReconcileOutcome retention on the runtime (lastOutcome(provider)) so a future status route (U19) can surface the last-cycle distilled/unchanged/failed/unavailable counts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t/webhook states (U18 follow-up) Widens SourceStatusSummary with a state field (pending/current/failed/ unavailable/over_limit), EnrollmentStatusSummary with collection + a per-state counts breakdown, and ProviderWebhookStatus.active with earliestExpiry — all derived purely from existing ProviderState fields, no discover/fetch/reconcile changes. Lets U19's status route and U20's UI render §13's actual state taxonomy without reimplementing per-adapter derivation outside the adapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… status precedence (U18 follow-up) Extracts earliestSubscriptionExpiry() so ensureWebhook's channel-level expiresAt and describeStatus's webhook.active.earliestExpiry share one Date.parse-based minimum instead of the latter duplicating it as an unsafe lexical string compare (Graph doesn't guarantee same-timezone/precision ISO-8601 timestamps). Also derives SourceStatusState's zero-counts map from SOURCE_STATUS_STATES instead of a hand-listed object literal, and adds a test pinning that available:false takes precedence over failed/over_limit when a source carries both a lastFailure and available:false. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… adapter factory wiring (U19)
Wires the provider-neutral enrollment/status HTTP routes to the U17/U18
adapter methods: POST /enrollments/preview, POST /enrollments, GET /status,
and DELETE /enrollments/{id}, each 404ing when the dispatched adapter method
is absent (Google/Notion untouched). Enrollment routes gate on
manage_integrations + CSRF + the provider's collection allowlist +
canWrite(role, collection), and the full enroll additionally refuses without
an explicit audience acknowledgement (R33). Enrollment persists into
ProviderState.enrollments via the existing encrypted-state read/lock/write
path; unenroll appends an `unenrolled` review event per source while
retaining source metadata (R38).
Threads the resolved user/role through IntegrationRouteAuthorization so
routes' ctx.user/ctx.role and the canWrite gate work, and wires RBAC roles +
distill.estimated_usd_per_call into the Microsoft adapter factory so
production previews compute real readers/ratifiers and a USD estimate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ility guard (U19 follow-up) Quality-review fixes on 8c51040: - DELETE no longer silently discards appendUnavailableReview's Result. A failed audit write (disk-full/permission) is now surfaced through the runtime's existing onError channel — the enrollment removal itself still succeeds (204), it just no longer loses the failure with nowhere to go. Threaded a new optional IntegrationRouteDependencies.onError, wired from runtime.ts's handle() (the same onError already used by the reconcile cycle) through to routes.ts. - DELETE gained the explicit adapter.resolveEnrollment === undefined capability guard the other three routes already had, so Google/Notion 404 up front instead of falling through into real manage_integrations + CSRF work and 404ing only incidentally on a missing enrollment id. - Added tests: malformed-JSON body -> clean 400 on preview and enroll, DELETE 404s before auth/CSRF for a capability-less adapter, and a failed review write surfaces via onError without failing the delete. - DRY'd the preview/POST-enroll shared gate sequence (parse body -> collection allowlist 422 -> canWrite 403 -> includeSpeakerNotes default) into one parseEnrollmentRequest() helper used by both routes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ CSP (U20)
Serves GET /integrations/microsoft/ui (server-rendered picker page: CSP,
collection dropdown, audience-ack-gated Enroll button, no inline script) and
GET /integrations/microsoft/ui/assets/{file} (vendored @azure/msal-browser +
msal-common browser build, plus the hand-authored glue.js driver and
picker-serializer.js), wired into routes.ts alongside U19's enrollment
routes. The page's own CSP (default-src/script-src 'self', connect-src
scoped to login.microsoftonline.com + *.sharepoint.com, form-action scoped to
SharePoint) is set per-response since serve sets none globally.
serializePickerSelection (picker-serializer.js) is the enforced choke point
for "the SharePoint token never crosses to Daftari": it allowlists
driveId/itemId/name/sharepointIds out of a raw File Picker v8 pick result and
drops everything else, so an accessToken/authentication field can never leak
through by omission. It's a plain ES module served as-is to the browser and
imported directly by the vitest suite — no duplicate implementation to drift.
@azure/msal-browser is dev-only (never imported by server code — it's
build-only tooling for producing the vendored bundle) and browser-only
(vendored dist/dist-browser output, patched to replace its one bare
`@azure/msal-common/browser` import with a relative path, so the whole
graph resolves via native ESM relative imports with no bundler). Vendored
assets live under src/integrations/microsoft/assets/, added to package.json's
`files` allowlist alongside dist/templates so they ship in a published
package.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… for msal silent auth (U20 follow-up) CRITICAL: glue.js's window "message" handler checked event.source === popup but not event.origin. source is the popup's window handle and survives a navigation of that window, so if the popup were ever navigated away from pickerHost (open redirect on the SharePoint host, injected third-party script in that origin), a forged message would still pass the source-only check and could solicit the SharePoint token via a forged "authenticate" command, or forge a "pick" result. Now requires event.origin === new URL(pickerHost).origin at the window-message gate before accepting "initialize"/any command; the MessageChannel port traffic after handoff inherits that trust, so the window gate is the one place this needed fixing. Also: - CSP: added frame-src https://login.microsoftonline.com. msal-browser's acquireTokenSilent opens a hidden iframe against that host; with no frame-src it inherited default-src 'self' and was silently blocked, forcing every session into the interactive popup even with a valid SSO session. This extends design §5.1's CSP (which omitted frame-src) — worth reconfirming at the probe-5 live smoke. Updated the byte-for-byte CSP assertion in microsoft-ui.test.ts. - Added src/integrations/microsoft/assets/VENDORING.md documenting the pinned msal-browser version, the two entry points the module graph was walked from, which subtrees were never reachable (custom_auth, redirect_bridge/popup_relay export-subpath code, both packages' lib/CJS trees), and the exact bare specifier patched (@azure/msal-common/browser -> relative) — so a version bump is reproducible without re-deriving this from scratch. A bead tracks turning the one-off process into a committed script. - Popup-closed recovery: openFilePickerPopup now polls popup.closed and rejects (re-enabling the Select-files flow) if the user closes the popup by hand, instead of leaving the button stuck forever with no popup left to respond. - One-line comment at the ui/assets RegExp in routes.ts noting `provider` is already constrained to PROVIDER_NAMES by providerFrom()'s own regex, so it's safe to interpolate unescaped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Microsoft 365 section to docs/integrations.md mirroring the Google/Notion depth: scope, config keys, Entra app registration (Web + SPA redirects, delegated scopes per scope_profile), webhook subscription model, enrollment/picker operator flow, extraction limits, and local state/retention. Verified against src/utils/config.ts, src/integrations/ microsoft.ts, routes.ts, state.ts, and review.ts — no code changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Sep 6, 2026
The M365 connector vendors @azure/msal-browser under src/integrations/microsoft/assets/vendor/. Biome was linting these third-party bundles (258 errors on Node 20/22), failing the build matrix. Exclude the vendor tree the same way regression fixtures are excluded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reconcileProvider persists the observed revision with an empty contentHash before distilling, so a failure retries like a new source. The distill-failure paths that record lastFailure spread ...next, which carries previous?.contentHash — for an *updated* source that reverts the hash to the prior non-empty value, and the next cycle's skip-guard then treats the unprocessed revision as already done. Force contentHash to "" on both failure paths, preserving the pending invariant while still recording the failure reason. Surfaced by main's new 'retries an updated source after a %s failure from persisted state' test once merged into this branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pdfjs-dist calls Promise.withResolvers, added only in Node 22. daftari supports Node 20 (engines: >=20.9.0, CI matrix includes 20), where it is absent, so getDocument() throws a TypeError and every PDF extraction fails — in-process and, once the worker loads, in the compiled worker too. Install the standard guarded polyfill at the top of the pdf extractor's module body, before any pdfjs call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…itation) The 13 tests that spawn the real worker_threads worker with a .ts entry via tsx can't run on Node 20: Node 22 propagates a --import-registered loader to workers, Node 20 does not, so the .ts worker entry fails with ERR_UNKNOWN_FILE_EXTENSION. This is a dev/test-only limitation — the published package ships compiled .js workers (execArgv []), which load on every supported Node version, and the withResolvers polyfill covers pdfjs on Node 20 there. Gate exactly those tests on a shared canRunTsWorkers predicate so the CI Node 20 leg is green while the same tests still run end-to-end on Node 22. The refusal/error paths in the U15 fetch suite (which never reach extraction) stay unskipped and keep Node 20 coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the Microsoft 365 ingestion connector designed in #492 / PR #501, as a third
ProviderAdapteron the existing integration layer plus a provider-neutralsrc/extract/and a browser picker page.21 implementation units (
docs/superpowers/plans/2026-09-02-m365-ingestion.md), each spec- and quality-reviewed, plus a final cross-unit review. ~13.9K insertions, 4993 tests green.What's here
PROVIDER_NAMESclosed-union refactor, optional provider-neutral state extensions, distill-collection override (default byte-identical), engine failure/reconnect persistence (terminal-only), webhook challenge/lifecycle route plumbing.src/extract/): boundedworker_threadsharness + native OOXML (docx/pptx, speaker notes preserved) +pdfjs-disttext-layer PDF, with zip-bomb/entity-expansion/encrypted guards and an evaluation corpus.src/integrations/microsoft.ts): Entra OAuth (scope-profile, rotated refresh, terminal-signal), Graphdeltadiscovery over the opaque-cursor contract, fetch/download (no token leak on the SAS redirect, 25 MiB cap, legacy→PDF), webhook subscriptions (timing-safe verify, one per drive), enrollment resolve/estimate,describeStatus.canWrite+ audience-ack gates), the File Picker v8 page (vendored msal, exact CSP, no token to Daftari, origin-validated postMessage), operator docs.Design decisions carried from #492
Known follow-ups (bead-tracked, none block the core flow)
enrollmentIdengine plumbing — until discovery tags sources with their enrollment, per-enrollment/statuscounts read 0 and unenroll review-events don't fire (sources still go unavailable via not-returned). In progress./enrollments/previewestimate in the picker so the cost/readers/ratifiers disclosure is visible before the human acknowledges. In progress.converted_unavailablereason wire-up, NotionequalSecretdedup, vendor-prune script.🤖 Generated with Claude Code