test: record doctor FTS feasibility

dev/plans/260723-doctor-single-pass-fts-feasibility.md

@@ -0,0 +1,139 @@
+# Prove a single-pass doctor FTS comparison before changing production
+
+## Goal
+
+Determine whether exact doctor FTS instance comparison can use one monotonic
+actual stream and one monotonic expected stream instead of retained term ranges
+plus repeated bidirectional `EXCEPT` queries.
+
+The accepted
+[document-interval feasibility result](../../docs/research/doctor-document-interval-feasibility.md)
+rejected document-ID-bounded actual-vocabulary scans because 41
+production-shaped intervals made dominant work 3.80 times slower. The current
+production path in
+`src/infrastructure/sqlite/fts-projection.ts:ftsProjectionSemanticContentIsValidReadOnly`
+still builds one complete, corpus-sized, memory-only expected FTS projection,
+streams actual and expected row vocabularies into a retained
+`DoctorTermRange[]`, then runs two instance `EXCEPT` queries per range.
+
+This plan is an evidence gate, not a production executor plan. Its candidate
+keeps the complete expected projection and existing exact docsize proof, so it
+may reduce comparison CPU and make comparison-side state constant but cannot
+claim bounded total memory. Production doctor remains unchanged until generated
+evidence passes, the reliance on FTS5 vocabulary order is explicitly accepted,
+and a separate production plan is approved.
+
+## Changes
+
+1. Extend `scripts/measure-doctor.ts` with a separate single-pass controller
+   mode, exposed as `pnpm measure:doctor:single-pass` in `package.json`. Reuse
+   the existing generated production-writer corpora, owned temporary root,
+   controller-issued worker authority, immutable clones, fresh child processes,
+   alternating strategy order, aggregate-only report, and success/failure
+   cleanup. Add full-only generated cohorts for a mixed ordinary-term index
+   whose current algorithm reports at least three production-sized term ranges
+   and for one repeated term above the 1,000,000-instance target. Keep the
+   existing `pnpm measure:doctor` arguments, corpora, and exact report unchanged.
+2. Put candidate query and comparison code in a controller-independent
+   `scripts/doctor-single-pass-probe.ts` whose effects are limited to
+   connection-local memory-only TEMP state with complete cleanup. The controller
+   and deterministic tests must use this one measurement-only implementation.
+   Compare byte-identical clones with three strategies:
+   - the current production
+     `ftsProjectionSemanticContentIsValidReadOnly` result and total elapsed
+     time;
+   - a term-ordered direct-stream candidate; and
+   - a fully ordered
+     `ORDER BY term COLLATE BINARY, doc, col COLLATE BINARY, offset` control
+     that reveals the cost and TEMP plan of portable explicit ordering.
+     The candidate and control must build the same complete expected TEMP FTS and
+     perform the same bidirectional docsize proof before instance comparison.
+3. For the candidate, prepare one actual and one expected
+   `fts5vocab(..., 'instance')` iterator ordered only by
+   `term COLLATE BINARY`. Decode SQLite integers as `bigint`. Validate every
+   coordinate as a non-empty term, signed 64-bit document ID, exact `text`
+   column, and non-negative signed-64-bit offset. Require each stream to be
+   strictly increasing by UTF-8 binary term, signed document ID, binary column,
+   then offset. Compare coordinates one-for-one and require both iterators to
+   reach EOF together. A malformed, duplicate, non-monotonic, unequal, missing,
+   or extra row fails closed.
+4. Retain only the current and previous coordinate from each stream. Close both
+   iterators through the existing
+   `src/infrastructure/sqlite/iterator-cleanup.ts:closeSqliteIterators`
+   error-precedence pattern before dropping TEMP objects. Keep
+   `PRAGMA temp_store = MEMORY`, require the read-back value `2`, preserve the
+   private savepoint and any outer transaction, and make load, iteration,
+   comparison, rollback, release, or drop failure unhealthy. No comparison
+   term, range, document, position, count map, hash, or array may grow with the
+   corpus.
+5. Capture normalized `EXPLAIN QUERY PLAN` facts and exact operation counters
+   for all strategies. Candidate admission requires exactly one actual and one
+   expected instance-vocabulary scan, no `EXCEPT`, no grouped summary query, no
+   TEMP B-tree, exact visited-row counts, and constant two-row lookahead.
+   Explicit full ordering is a control, not an acceptable fallback if it
+   materializes corpus-sized sort state. The real production baseline reports
+   only total elapsed time; do not invent unavailable baseline phases or modify
+   `src/` to expose them. Candidate and control phases are expected-index load,
+   docsize, instance comparison, cleanup, and total. Peak RSS remains supporting
+   evidence rather than a threshold.
+6. Add `test/doctor-single-pass-measurement.test.ts` with two distinct proofs:
+   - import the measurement-only probe and require the same healthy/unhealthy
+     decisions as the current complete-vocabulary oracle for empty projections,
+     zero-token rows, sparse signed IDs including integer extrema, multibyte and
+     canonically distinct Unicode, missing or extra first/middle/final
+     coordinates, equal-count wrong terms, shifted documents/columns/offsets,
+     cross-document swaps, and docsize damage. Keep the routine contract corpus
+     bounded and assert constant lookahead with a reduced repeated-term target;
+     add a fixed-seed small-corpus matrix whose healthy and damaged cases must
+     agree with the oracle; and
+   - spawn the reduced controller contract and require recursive output
+     allowlisting, exact aggregate derivation, fixed query/row counters,
+     private modes, unchanged database bytes and metadata, absent WAL/SHM/
+     journal artifacts, silent private-worker failure, and complete cleanup.
+     Exercise different healthy FTS segment topologies on equal content so the
+     monotonicity guard is not proved by one insertion shape.
+7. Run the reduced ordering contract on the minimum supported Node runtime and
+   the current release runtime. Record Node and SQLite versions. The
+   [public `fts5vocab` contract](https://www.sqlite.org/fts5.html#the_fts5vocab_virtual_table_module)
+   exposes instance coordinates but does not promise the complete native tuple
+   order used by the candidate. A future runtime that changes that order must
+   fail a healthy library closed through the monotonicity guard; a production
+   plan may accept that compatibility dependency only through an explicit human
+   decision.
+8. Record the result in
+   `docs/research/doctor-single-pass-fts-feasibility.md`. Require exact
+   correctness, immutability, privacy, cleanup, and structural work assertions
+   before interpreting time. The opt-in full measure must cover the existing
+   large high-unique-term cohort, the mixed corpus with at least three observed
+   baseline ranges, and the dominant repeated term crossing the current
+   1,000,000-instance target. Run at least two complete invocations and report
+   median and p95 total ratios per cohort plus candidate phase scaling. A
+   candidate slower than the real production baseline in any named large cohort
+   is rejected. Non-regressive but noise-sized or inconsistent gains are
+   inconclusive, not acceptance. There is no automatic elapsed-time pass:
+   explicit human review decides whether repeated gains are meaningful and
+   whether the ordering dependency is acceptable before a separate production
+   executor plan may be created.
+
+## Verify
+
+- `pnpm test test/doctor-single-pass-measurement.test.ts`
+- `pnpm measure:doctor:single-pass`
+- `pnpm check`
+
+## Boundaries
+
+- Do not change `src/` production doctor behavior, public health/CLI contracts,
+  output, exit codes, provider reads, or persistent library state in this
+  feasibility slice.
+- Do not describe direct instance streaming as bounded total memory. The full
+  expected TEMP FTS remains corpus-sized.
+- Do not revive document-ID vocabulary scans, spill transcript-derived terms or
+  coordinates to disk, reimplement `unicode61`, load a native extension, open
+  the library writable for FTS5 `integrity-check`, sample evidence, or replace
+  exact coordinates with counts or hashes.
+- Stop if deterministic ordering needs a TEMP B-tree, either vocabulary is
+  scanned more than once, comparison-side retained state grows with terms or
+  instances, any corruption decision differs, a hard immutability/privacy
+  assertion fails, or the candidate regresses any named large cohort. Treat
+  noise-sized or inconsistent gains as inconclusive and keep the current audit.

dev/plans/README.md

@@ -20,8 +20,12 @@ evidence-gated slices receive an executor plan only after their gate passes.
    replacement work, remove avoidable provider lookup/serialization and
    statement-per-content costs, then gate deeper transaction or WAL changes.
 2. [Doctor and maintenance hot paths](260723-doctor-maintenance-hot-paths.md) —
-   make exact FTS verification memory-bounded, partition oversized terms, page
-   only orphan candidates, and gate any compact-proof optimization separately.
-3. [Verified bounded session reads](260723-verified-bounded-session-reads.md) —
+   record the rejected document-interval gate, page only orphan candidates, and
+   keep compact-proof work separately gated.
+3. [Single-pass doctor FTS feasibility](260723-doctor-single-pass-fts-feasibility.md) —
+   evaluate an exact term-ordered alternative without repeated
+   actual-vocabulary scans; gate any production refactor on corruption parity,
+   memory-only state, compatible ordering, and measured scaling.
+4. [Verified bounded session reads](260723-verified-bounded-session-reads.md) —
    stream complete validation and the existing public-document digest while
    retaining only the requested bounded `show` or `export` selection.

docs/architecture-memo.md

@@ -1331,6 +1331,18 @@ canonical reads, batches the TEMP FTS load transactionally, partitions exact FTS
 instance comparison by a fixed occurrence target, reports interactive phases,
 and supports aggregate opt-in timings without weakening its exact proof. The
 complete expected index and one oversized term remain corpus-dependent.
+
+Generated feasibility evidence rejected document-ID-bounded actual-vocabulary
+scans: 41 production-shaped intervals made total work 3.80 times slower and
+grouped term-summary work 6.82 times slower while retaining the same
+virtual-table scan shape. The accepted decision preserves the current exact
+whole-library audit and retires that interval design. A separate
+[single-pass feasibility plan](../dev/plans/260723-doctor-single-pass-fts-feasibility.md)
+may evaluate one monotonic actual/expected instance traversal as a CPU
+optimization, but no production refactor is authorized until it proves exact
+corruption parity, memory-only transient state, compatible ordering, and better
+scaling. It does not solve the corpus-sized expected index.
+
 Crash-safe index generations now use a transaction-bound schema-3 receipt to
 replace global recovery scans only at exact certified boundaries; any ambiguity
 keeps the complete validation fallback. Further lower total-memory doctor work,

docs/contributing/commands.md

@@ -11,6 +11,7 @@
 | `pnpm format:docs:check`                                | Check Markdown formatting                                               | No                                 | No                                  |
 | `pnpm lint` / `pnpm lint:fix`                           | Check or fix source/test lint                                           | Fix variant only                   | No                                  |
 | `pnpm measure:content-storage`                          | Compare legacy and current canonical-content layouts with fixed corpora | Temporary directory, removed       | No                                  |
+| `pnpm measure:doctor`                                   | Probe document-bounded exact FTS validation with generated libraries    | Temporary directory, removed       | No                                  |
 | `pnpm measure:query-lineage`                            | Compare repeated and query-scoped lineage resolution                    | No                                 | No                                  |
 | `pnpm measure:entry-query`                              | Measure fixed textless entry queries through production SQLite seams    | In-memory database, removed        | No                                  |
 | `pnpm measure:search-query`                             | Measure a fixed broad first-page search through production SQLite seams | In-memory database, removed        | No                                  |
@@ -44,6 +45,18 @@ checks require exact-content ID reuse, collision coexistence, no target index on
 text, and a realistic target database no larger than 60% of the legacy layout.
 Timings are report-only and can vary by machine and SQLite version.
 
+`pnpm measure:doctor` is an opt-in feasibility probe outside `pnpm check`. It
+seeds two provider-free libraries through the production writer, measures one,
+two, and many exact document-bounded FTS vocabulary strategies in alternating
+fresh child processes, and runs exact equality and final-health checks outside
+the measured workers. The many-interval strategy uses the proposed 512-row and
+16-MiB admission limits. It reports only generated aggregate plan, timing,
+memory, corpus, and integrity evidence. Immutable file state, absent sidecars,
+owned permissions, and complete success/failure cleanup are required; elapsed
+time and RSS are report-only. The probe is supported on macOS and Linux because
+the private-file evidence requires POSIX modes. The result cannot authorize a
+production refactor without explicit human acceptance.
+
 `pnpm measure:query-lineage` is also opt-in and outside `pnpm check`. It compares
 rebuilding lineage state per resolution with one query-scoped resolver over a
 deterministic generic in-memory corpus. Exact result equality is required;

docs/contributing/testing.md

@@ -204,6 +204,19 @@ composition, forced load failure, cleanup, and exact semantic mismatches.
 Application, runtime, and CLI tests keep progress/timing best-effort, normal
 redirected stderr quiet, and doctor stdout/check/exit behavior unchanged.
 
+`pnpm measure:doctor` is the opt-in, provider-free document-interval feasibility
+probe. It seeds small and large generated libraries through the production
+writer, measures one, two, and many document-bounded `fts5vocab` strategies in
+alternating fresh child processes, and records normalized plan facts, elapsed
+phases, and peak RSS. The many strategy uses the proposed 512-row and 16-MiB
+admission limits. Separate exact comparisons require complete coordinate,
+term-summary, docsize, interval, tail, final-health, immutable-file, sidecar,
+permission, and cleanup equality. A reduced `--contract` mode owns deterministic
+script, direct-worker isolation, and forced-failure coverage in Vitest on macOS
+and Linux; Windows skips the POSIX-mode contract. Timings never pass or fail CI,
+and the measurement cannot authorize a production doctor refactor; its recorded
+result requires explicit human acceptance.
+
 An accepted live doctor measurement may run against the ordinary retained
 library only as one process: do not open the database through `sqlite3` or any
 second SQLite connection while the immutable snapshot is active. Capture the