diff --git a/.claude/skills/e2e-from-prs/SKILL.md b/.claude/skills/e2e-from-prs/SKILL.md index 5770468..25a1cd2 100644 --- a/.claude/skills/e2e-from-prs/SKILL.md +++ b/.claude/skills/e2e-from-prs/SKILL.md @@ -31,8 +31,7 @@ or code the e2e user cannot reach. Also skip when an existing test already asser ## 3. Write at most 3 tests -1. Read `references/playwright.md` (next to this file), every file in `e2e/helpers/`, and the whole - target spec before you write anything. +1. Read every file in `e2e/helpers/` and the whole target spec before you write anything. 2. Pick the spec by area: `agent`, `ask`, `tree` (sections), `crop` (figures, pages), `publish` (wiki), `upload`, `parse` (import, sectionize, jobs). Never add to `smoke.spec.ts`. Create `e2e/specs/.spec.ts` only if no area fits. @@ -50,6 +49,7 @@ Rules: - Data you create is titled `` `${PREFIX} W ${Date.now()}` `` and deleted in `afterAll` (`deleteByPrefix`, `deleteTestProjects`, `deleteImport`). Never delete `[fixture]` records. A test that changes a fixture restores it in `afterEach` (see `restoreTree` in `tree.spec.ts`). +- Make test data in code (`helpers/pdf.ts`, API calls); never add files. - Wait on database state with `waitForValue` or `waitFor` through `api`. Never wait on UI progress text, and never use `page.waitForTimeout`. - `@llm` tests: `test.describe.configure({ retries: 1 })`, assert records through `api`, never the diff --git a/.claude/skills/e2e-from-prs/references/playwright.md b/.claude/skills/e2e-from-prs/references/playwright.md deleted file mode 100644 index 63fc004..0000000 --- a/.claude/skills/e2e-from-prs/references/playwright.md +++ /dev/null @@ -1,126 +0,0 @@ -# Playwright suite conventions - -The orchestrator picks the location while writing the plan (shown at the plan checkpoint), -then builds the config and helpers in pre-flight. Test agents only add specs. - -| The app has | `` (suite) | `` (package.json, run commands here) | `` | -|---|---|---|---| -| UI tests already (`cypress/`, `e2e/`, `tests/ui/`, ...) | Ask the user: extend that framework, or add Playwright beside it. Never add a second framework silently | | | -| `frontend/package.json` (Vue SPA) | `/frontend/e2e/` | `/frontend` (reuse its `node_modules`) | `-c e2e` | -| Neither (Desk-only or portal app) | `/e2e/` | `/e2e` (own small `package.json`) | `-c .` | - -Playwright tests Desk (`/app/...`) and portal pages the same way. - -## Layout - -``` -/ - playwright.config.ts - .env.e2e E2E_BASE_URL, E2E_USER, E2E_PASSWORD, E2E_API_KEY, E2E_API_SECRET - .gitignore .env.e2e .state/ test-results/ playwright-report/ - helpers/ - env.ts reads the E2E_* vars; PREFIX = "[test]" - api.ts call(method, args), getValue, getList (token auth) - wait.ts waitForValue(doctype, name, field, expected, { timeout, interval }) - upload.ts uploadViaUi(page, input, path); uploadFile(path, { doctype, docname }) - cleanup.ts deleteByPrefix(doctype, titleField, prefix) - test.ts `test` extended with `api` and `fixture` (reads .state/fixtures.json) - setup/ - auth.setup.ts log in once, save .state/user.json - seed.setup.ts find-or-create the [fixture] records, wait for their final status, - write .state/fixtures.json - specs/ - .spec.ts one file per area, test titles start with the plan's test ID -``` - -## Config essentials - -- Load the env file first: `process.loadEnvFile("")` (Node 20.12+). The path is - relative to ``, where every command runs: `e2e/.env.e2e` in a SPA, `.env.e2e` - standalone. On older Node use the `dotenv` package: `dotenv.config({ path: "" })`. -- `testDir: "./specs"`, `use.baseURL` from `E2E_BASE_URL`, `use.storageState: ".state/user.json"`. -- Two projects: `setup` (matches `setup/*.setup.ts`, no storageState) and `chromium` - (`dependencies: ["setup"]`). -- `workers: 1` while specs share the fixture. Raise it only for specs that create their own data. -- Long jobs: per-test `test.setTimeout(...)`; don't raise the global timeout for everyone. - -## Helpers contract - -- **Auth:** `auth.setup.ts` posts `/api/method/login` (`usr`, `pwd`) with the `request` fixture - and saves `request.storageState()`. UI specs start logged in. -- **API calls use token auth** (`Authorization: token :`). A browser session that - has loaded a page has a CSRF token, and cookie POSTs without `X-Frappe-CSRF-Token` fail; - token auth avoids that. Make keys in pre-flight: - `bench --site execute frappe.core.doctype.user.user.generate_keys --args "['']"`. - It rotates the secret, so use a user nothing else depends on. Some Frappe versions return - only `api_secret`; then read the key with - `bench --site execute frappe.db.get_value --args "['User', '', 'api_key']"`. - Write both into `.env.e2e`. -- **`call` throws** on a non-2xx response with the server message; returns `message` otherwise. -- **`waitForValue` polls the API** (default every 3 s) and fails with the last value seen plus - the record's error field. Never wait on UI progress text; it can be stale. -- **Seed:** `seed.setup.ts` looks up `[fixture] ` first and creates it only if it is - missing or not in its final status. Expensive pipelines run once per site, not per run. -- **Data:** every record a spec creates is titled `` `${PREFIX} ${Date.now()}` `` - and deleted in `afterAll` with `deleteByPrefix`. Specs never delete `[fixture]` records; a - spec that changes one restores it in `afterEach`, or creates its own `[test]` record instead. -- **Selectors:** `getByRole`, `getByLabel`, `getByText`, matching the case file's "button - 'Start'" wording. Dialogs: `page.getByRole("dialog")`. No CSS classes from component - libraries, no `nth()` unless the case says "the second image". - -## Tags - -Every test has exactly one tier tag (`@smoke`, `@functional` or `@negative`) and one area tag. -Add `@sanity` to the 1-3 fastest tests that prove an area still works. Add `@llm` on top of the -tier tag when the test needs the AI (`@functional @llm @agent`), so AI tests still run in tier -runs and can also run on their own. Apps with no AI features have no `@llm` tag. - -| Tag | Purpose | Size / time | When | -|---|---|---|---| -| `@smoke` | Main journey works at all | under 5 min | after every deploy or build | -| `@sanity` | One area after a fix (combine with `@`) | 1-3 tests per area | after a fix | -| `@functional` | Every feature with good input | the full set | before a release | -| `@negative` | Bad and boundary input fails cleanly | | before a release | -| `@llm` | Extra tag, AI apps only: needs the AI, real configured model | | alone on demand; costs credits | - -```ts -import { test, expect } from "../helpers/test"; - -test("F-TREE-01 rename persists after reload", { tag: ["@functional", "@tree", "@sanity"] }, async ({ page, api, fixture }) => { - ... -}); -``` - -`@llm` rules: put `test.describe.configure({ retries: 1 })` in the describe block; assert DB -state through `api`, never the reply text; never override the site's model setting. - -## Known failures - -A test that FAILED in the exploratory run is kept, marked expected-to-fail: - -```ts -test("F-AGENT-11 stop ends the turn", { - tag: ["@functional", "@agent"], - annotation: { type: "issue", description: "https://github.com///issues/" }, -}, async ({ page }) => { - // known failure: #. Remove test.fail() when the issue is closed. - test.fail(); - ... -}); -``` - -When the bug is fixed the test passes, Playwright reports "expected to fail, but passed", and -someone removes `test.fail()`. A spec that can't be made reliable gets `test.fixme()` with the -reason; it doesn't count as coverage. - -## Commands (from ``) - -```bash - add -D @playwright/test && npx playwright install chromium # once; = yarn or npm, per the lockfile -npx playwright test --project setup # seed fixture + login -npx playwright test --grep @smoke # after deploy/build -npx playwright test --grep "(?=.*@sanity)(?=.*@tree)" # after a tree fix -npx playwright test # before a release (metered service: check balance) -npx playwright test --grep @llm # AI apps only: AI tests, on demand -npx playwright show-report /playwright-report -``` diff --git a/.github/actions/e2e-site/action.yml b/.github/actions/e2e-site/action.yml index 1781484..de518d0 100644 --- a/.github/actions/e2e-site/action.yml +++ b/.github/actions/e2e-site/action.yml @@ -95,7 +95,8 @@ runs: working-directory: /home/runner/frappe-bench run: | nohup bench serve --port 8000 --noreload > logs/e2e-web.log 2>&1 & - nohup bench worker > logs/e2e-worker.log 2>&1 & + # With CI set, frappe runs every job in test mode, which turns Local models off. + nohup env -u CI bench worker > logs/e2e-worker.log 2>&1 & nohup node apps/frappe/socketio.js > logs/e2e-socketio.log 2>&1 & for attempt in $(seq 1 60); do curl -sf http://wikify.localhost:8000/api/method/ping && exit 0 diff --git a/.github/workflows/agent-ci-probe.yml b/.github/workflows/agent-ci-probe.yml new file mode 100644 index 0000000..2f6bcbf --- /dev/null +++ b/.github/workflows/agent-ci-probe.yml @@ -0,0 +1,53 @@ +name: Agent CI probe + +on: + pull_request: + paths: + - .github/workflows/agent-ci-probe.yml + - wikify/agent_ci_probe.py + +permissions: + contents: read + +jobs: + probe: + runs-on: ubuntu-latest + timeout-minutes: 30 + env: + CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + services: + redis-cache: + image: redis:alpine + ports: + - 13000:6379 + redis-queue: + image: redis:alpine + ports: + - 11000:6379 + mariadb: + image: mariadb:11.8 + env: + MYSQL_ROOT_PASSWORD: root + ports: + - 3306:3306 + options: --health-cmd="mariadb-admin ping" --health-interval=5s --health-timeout=2s --health-retries=3 + steps: + - uses: actions/checkout@v7 + - uses: ./.github/actions/e2e-site + - name: Direct turn + shell: bash + working-directory: /home/runner/frappe-bench + run: bench --site wikify.localhost execute wikify.agent_ci_probe.direct + - name: Ask + if: always() + shell: bash + working-directory: /home/runner/frappe-bench + run: bench --site wikify.localhost execute wikify.agent_ci_probe.ask + - name: Worker turn + if: always() + shell: bash + working-directory: /home/runner/frappe-bench + run: | + bench --site wikify.localhost execute wikify.agent_ci_probe.worker + echo "----- worker log tail" + tail -n 80 logs/e2e-worker.log diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 31598ac..79927ba 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -9,11 +9,15 @@ on: - .github/workflows/e2e.yml - .github/actions/e2e-site/** +defaults: + run: + shell: bash + jobs: playwright: name: Playwright runs-on: ubuntu-latest - timeout-minutes: 240 + timeout-minutes: 300 permissions: contents: read issues: write diff --git a/frontend/e2e/.env.e2e.example b/frontend/e2e/.env.e2e.example index ef8ca7f..fdd2b21 100644 --- a/frontend/e2e/.env.e2e.example +++ b/frontend/e2e/.env.e2e.example @@ -3,3 +3,4 @@ E2E_USER=e2e@example.com E2E_PASSWORD= E2E_API_KEY= E2E_API_SECRET= +# E2E_FIXTURE_PDF=../docs/what-is-wikify.pdf # small doc for local runs until the bench has the 6 h parse timeout diff --git a/frontend/e2e/fixtures/README.md b/frontend/e2e/fixtures/README.md new file mode 100644 index 0000000..4d3089a --- /dev/null +++ b/frontend/e2e/fixtures/README.md @@ -0,0 +1 @@ +Adapted from *Anatomy and Physiology*, J. Gordon Betts et al., OpenStax (2013), pp. 1155–1252, CC BY 4.0, https://openstax.org/details/books/anatomy-and-physiology. Changes: page range extracted; three University of Michigan micrographs (CC BY-NC-SA) removed. diff --git a/frontend/e2e/fixtures/openstax-anatomy-ch24-25.pdf b/frontend/e2e/fixtures/openstax-anatomy-ch24-25.pdf new file mode 100644 index 0000000..b083c18 Binary files /dev/null and b/frontend/e2e/fixtures/openstax-anatomy-ch24-25.pdf differ diff --git a/frontend/e2e/helpers/cleanup.ts b/frontend/e2e/helpers/cleanup.ts index 938bd83..5f0aba5 100644 --- a/frontend/e2e/helpers/cleanup.ts +++ b/frontend/e2e/helpers/cleanup.ts @@ -1,5 +1,6 @@ import type { Api } from "./api"; import { waitFor } from "./wait"; +import { renumberTree } from "./wikify"; const BUSY = ["Queued", "Parsing", "Remediating", "Generating Wiki"]; @@ -51,17 +52,32 @@ export async function waitForIdle( ).catch((error) => console.warn(`cleanup: ${error.message}`)); } +// A REST delete runs NestedSet's table-wide lft shift, which reorders other documents' trees +// (the [fixture] source among them). delete_section deletes raw and renumbers only this document. +async function deleteSections(api: Api, sourceDocument: string): Promise { + const filters = { source_document: sourceDocument, parent_source_section: ["is", "not set"] }; + const roots = await api.getList("Source Section", { filters }); + if (roots.length) await renumberTree(api, sourceDocument); + for (const root of roots) { + await api + .call("wikify.api.sections.delete_section", { name: root.name }) + .catch((error) => console.warn(`cleanup: delete_section ${root.name}: ${error.message}`)); + } + await deleteTree(api, "Source Section", { source_document: sourceDocument }); +} + export async function deleteImport(api: Api, importName: string): Promise { await waitForIdle(api, importName); const imp = await api.getValue("Wikify Import", importName, ["source_document"]); const sourceDocument = imp?.source_document; await deleteByFilter(api, "Import Log Entry", { import: importName }); if (sourceDocument) { + // Unlinked first: delete_section refuses a document whose import is Completed. + await api.setValue("Wikify Import", importName, { source_document: null }); await deleteByFilter(api, "Wikify Agent Session", { source_document: sourceDocument }); - await deleteTree(api, "Source Section", { source_document: sourceDocument }); + await deleteSections(api, sourceDocument); await deleteByFilter(api, "Section Reference", { source_document: sourceDocument }); await deleteByFilter(api, "Source Page", { source_document: sourceDocument }); - await api.setValue("Wikify Import", importName, { source_document: null }); await api.setValue("Source Document", sourceDocument, { wiki_root_group: null, wiki_space: null, diff --git a/frontend/e2e/helpers/env.ts b/frontend/e2e/helpers/env.ts index f68274c..05b5b14 100644 --- a/frontend/e2e/helpers/env.ts +++ b/frontend/e2e/helpers/env.ts @@ -21,4 +21,13 @@ export const PREFIX = "[test]"; export const FIXTURE_PREFIX = "[fixture]"; export const STATE_DIR = path.resolve(import.meta.dirname, "../.state"); -export const FIXTURE_PDF = path.resolve(import.meta.dirname, "../../../docs/what-is-wikify.pdf"); +export const WHAT_IS_WIKIFY_PDF = path.resolve( + import.meta.dirname, + "../../../docs/what-is-wikify.pdf", +); +export const FIXTURE_PDF_OVERRIDDEN = !!process.env.E2E_FIXTURE_PDF; +export const FIXTURE_PDF = FIXTURE_PDF_OVERRIDDEN + ? path.resolve(import.meta.dirname, "../..", process.env.E2E_FIXTURE_PDF!) + : path.resolve(import.meta.dirname, "../fixtures/openstax-anatomy-ch24-25.pdf"); +export const FIXTURE_PDF_SHA256 = + "62d650cf40e02faea4395dc2301891c9e5165f88630713d4d92b7f704d0d6fcf"; diff --git a/frontend/e2e/helpers/test.ts b/frontend/e2e/helpers/test.ts index ca451eb..5086102 100644 --- a/frontend/e2e/helpers/test.ts +++ b/frontend/e2e/helpers/test.ts @@ -4,11 +4,21 @@ import { test as base } from "@playwright/test"; import { Api } from "./api"; import { STATE_DIR } from "./env"; -export type ImportFixture = { import: string; sourceDocument: string; title: string }; +export type OutlineEntry = { name: string; title: string; page_start: number; page_end: number }; + +export type FixtureOutline = { root: OutlineEntry | null; children: OutlineEntry[] }; + +export type ImportFixture = { + import: string; + sourceDocument: string; + title: string; + outline: FixtureOutline; +}; export type Fixtures = { project: string; projectName: string; + source: ImportFixture; review: ImportFixture; agent: ImportFixture; }; diff --git a/frontend/e2e/helpers/wikify.ts b/frontend/e2e/helpers/wikify.ts index 05b453f..944cac8 100644 --- a/frontend/e2e/helpers/wikify.ts +++ b/frontend/e2e/helpers/wikify.ts @@ -117,20 +117,23 @@ export async function pageRows(api: Api, sourceDocument: string): Promise }); } -export const FIXTURE_ROOT = "What Wikify Is"; -export const FIXTURE_SECTIONS = [ - "Where your documents live", - "How Wikify processes a PDF", - "Reading the page review", - "Sections and their labels", - "Finding content across documents", - "Generating the wiki", - "The assistant", - "Which AI service Wikify uses", -]; +// move_section rebuilds lft/rgt for the whole document from sort_order. +export async function renumberTree(api: Api, sourceDocument: string): Promise { + const [firstRoot] = await api.getList("Source Section", { + filters: { source_document: sourceDocument, parent_source_section: ["is", "not set"] }, + orderBy: "sort_order asc, lft asc", + limit: 1, + }); + if (!firstRoot) return; + await api.call("wikify.api.sections.move_section", { + name: firstRoot.name, + new_parent: "", + new_index: 0, + }); +} // A REST save on a section with an empty old_parent runs NestedSet's move and scrambles lft, so -// renumber the tree from sort_order afterwards (move_section rebuilds the whole document's tree). +// renumber the tree afterwards. export async function setSectionMarkdown( api: Api, sourceDocument: string, @@ -140,14 +143,153 @@ export async function setSectionMarkdown( for (const { name, markdown } of updates) { await api.setValue("Source Section", name, { markdown }); } - const [firstRoot] = (await sectionRows(api, sourceDocument)) - .filter((row) => !row.parent_source_section) - .sort((a, b) => a.sort_order - b.sort_order); - await api.call("wikify.api.sections.move_section", { - name: firstRoot.name, - new_parent: "", - new_index: 0, + await renumberTree(api, sourceDocument); +} + +const IMPORT_FIELDS = ["pdf", "page_count", "stage_label", "stage_progress", "started_at"]; +const DOCUMENT_FIELDS = [ + "pdf", + "page_count", + "parser_used", + "mean_score", + "canonical_mean", + "llm_cost", + "status", +]; +const PAGE_FIELDS = [ + "page_no", + "kind", + "image", + "verdict", + "composite", + "text_recall", + "extra_ratio", + "table_score", + "judge_score", + "llm_cost", + "notes", + "baseline_markdown", + "remediation_method", + "remediation_adopted", + "remediation_composite", + "remediation_notes", + "remediation_markdown", + "canonical_source", + "canonical_composite", + "canonical_markdown", +]; +const SECTION_FIELDS = [ + "title", + "is_group", + "section_type", + "level", + "sort_order", + "page_start", + "page_end", + "include_in_wiki", + "hierarchy_path", + "markdown", +]; +const REFERENCE_FIELDS = ["target_page", "anchor_text", "occurrences"]; + +function pick(row: Row, fields: string[]): Row { + return Object.fromEntries(fields.map((field) => [field, row[field]])); +} + +async function insertMany(api: Api, doctype: string, docs: Row[]): Promise { + const names: string[] = []; + for (let start = 0; start < docs.length; start += 50) { + const batch = docs.slice(start, start + 50).map((doc) => ({ doctype, ...doc })); + names.push(...(await api.call("frappe.client.insert_many", { docs: batch }))); + } + return names; +} + +export async function cloneImport( + api: Api, + sourceImport: string, + { title, project }: { title: string; project: string }, +): Promise<{ import: string; sourceDocument: string }> { + const original = await api.getValue("Wikify Import", sourceImport, [ + "source_document", + ...IMPORT_FIELDS, + ]); + const from = original.source_document; + const document = await api.getDoc("Source Document", from); + const sourceDocument: string = ( + await api.call("frappe.client.insert", { + doc: { doctype: "Source Document", title, project, ...pick(document, DOCUMENT_FIELDS) }, + }) + ).name; + + const pages = await api.getList("Source Page", { + filters: { source_document: from }, + fields: PAGE_FIELDS, + orderBy: "page_no asc", + }); + await insertMany( + api, + "Source Page", + pages.map((page) => ({ ...page, source_document: sourceDocument })), + ); + + // A child needs its parent's new name, so each pass inserts the rows whose parent is already copied. + let pending = await api.getList("Source Section", { + filters: { source_document: from }, + fields: ["name", "parent_source_section", ...SECTION_FIELDS], + orderBy: "lft asc", }); + const names = new Map(); + while (pending.length) { + const ready = pending.filter( + (row) => !row.parent_source_section || names.has(row.parent_source_section), + ); + if (!ready.length) throw new Error(`Orphan sections in ${from}: ${JSON.stringify(pending)}`); + const inserted = await insertMany( + api, + "Source Section", + ready.map((row) => ({ + ...pick(row, SECTION_FIELDS), + source_document: sourceDocument, + parent_source_section: names.get(row.parent_source_section) ?? null, + })), + ); + ready.forEach((row, index) => names.set(row.name, inserted[index])); + pending = pending.filter((row) => !names.has(row.name)); + } + await renumberTree(api, sourceDocument); + + const references = await api.getList("Section Reference", { + filters: { source_document: from }, + fields: ["from_section", "to_section", ...REFERENCE_FIELDS], + }); + await insertMany( + api, + "Section Reference", + references + .filter((row) => names.has(row.from_section) && names.has(row.to_section)) + .map((row) => ({ + ...pick(row, REFERENCE_FIELDS), + from_section: names.get(row.from_section), + to_section: names.get(row.to_section), + source_document: sourceDocument, + })), + ); + + const importName: string = ( + await api.call("frappe.client.insert", { + doc: { + doctype: "Wikify Import", + import_title: title, + project, + status: "Review", + source_document: sourceDocument, + ...pick(original, IMPORT_FIELDS), + }, + }) + ).name; + await api.setValue("Source Document", sourceDocument, { import: importName }); + return { import: importName, sourceDocument }; } export function escapeRegExp(text: string): string { @@ -171,14 +313,19 @@ export async function propagationPending(api: Api, sourceDocument: string): Prom return jobs.some((job) => JSON.parse(job.arguments).kwargs?.source_document === sourceDocument); } -// A fresh parse runs LLM remediation, so heading text and image captions vary between parses. -// Specs that parse their own import pick sections by position and figures by search. +// Two parses of the same PDF differ in headings, captions and even tree shape, so specs pick +// sections by position and figures by search, never by title or page number. export async function outline(api: Api, sourceDocument: string) { const rows = await sectionRows(api, sourceDocument); const topLevel = rows.filter((row) => !row.parent_source_section); - // Usually one title section holds the rest; a parse that misses the title heading comes out flat. - const root = topLevel.length === 1 ? topLevel[0] : undefined; - const children = root ? rows.filter((row) => row.parent_source_section === root.name) : topLevel; + const childrenOf = (parent: SectionRow) => + rows.filter((row) => row.parent_source_section === parent.name); + const widest = topLevel.reduce( + (best, row) => (!best || childrenOf(row).length > childrenOf(best).length ? row : best), + undefined, + ); + const root = widest && childrenOf(widest).length >= 5 ? widest : undefined; + const children = root ? childrenOf(root) : topLevel; if (children.length < 5) { const shape = rows.map((row) => [row.title, row.parent_source_section]); throw new Error(`Unexpected section tree for ${sourceDocument}: ${JSON.stringify(shape)}`); diff --git a/frontend/e2e/setup/seed.setup.ts b/frontend/e2e/setup/seed.setup.ts index 421a612..60602aa 100644 --- a/frontend/e2e/setup/seed.setup.ts +++ b/frontend/e2e/setup/seed.setup.ts @@ -1,73 +1,186 @@ +import { createHash } from "node:crypto"; import fs from "node:fs"; import { test as setup } from "@playwright/test"; import { Api } from "../helpers/api"; import { deleteImport } from "../helpers/cleanup"; -import { FIXTURE_PREFIX, STATE_DIR } from "../helpers/env"; -import { FIXTURES_FILE, type Fixtures, type ImportFixture } from "../helpers/test"; import { + FIXTURE_PDF, + FIXTURE_PDF_OVERRIDDEN, + FIXTURE_PDF_SHA256, + FIXTURE_PREFIX, + STATE_DIR, +} from "../helpers/env"; +import { + FIXTURES_FILE, + type FixtureOutline, + type Fixtures, + type ImportFixture, +} from "../helpers/test"; +import { + cloneImport, createProject, - FIXTURE_ROOT, - FIXTURE_SECTIONS, findProject, - PARSE_TIMEOUT, - sectionRows, + outline, + renumberTree, startImport, waitForImport, + type SectionRow, } from "../helpers/wikify"; const PROJECT_NAME = `${FIXTURE_PREFIX} e2e`; +const SOURCE_PROJECT_NAME = `${FIXTURE_PREFIX} source`; +const SOURCE_TITLE = `${FIXTURE_PREFIX} source`; const IN_PROGRESS = ["Queued", "Parsing", "Remediating"]; +const SOURCE_PARSE_TIMEOUT = 7_200_000; + +function readPrevious(): Partial { + try { + return JSON.parse(fs.readFileSync(FIXTURES_FILE, "utf8")); + } catch { + return {}; + } +} + +function entry({ name, title, page_start, page_end }: SectionRow) { + return { name, title, page_start, page_end }; +} + +function withoutNames({ root, children }: FixtureOutline): string { + const strip = ({ name, ...rest }: { name: string }) => rest; + return JSON.stringify({ root: root && strip(root), children: children.map(strip) }); +} + +async function intactOutline( + api: Api, + sourceDocument: string, + recorded?: FixtureOutline, +): Promise { + try { + const { rows, root, children } = await outline(api, sourceDocument); + if (rows.some((row) => !row.include_in_wiki)) return undefined; + const current = { root: root ? entry(root) : null, children: children.map(entry) }; + if (recorded && JSON.stringify(current) !== JSON.stringify(recorded)) return undefined; + return current; + } catch { + return undefined; + } +} + +async function findOrCreateProject(api: Api, projectName: string): Promise { + return (await findProject(api, projectName)) || (await createProject(api, projectName)); +} -async function isPristine(api: Api, sourceDocument: string): Promise { - const rows = await sectionRows(api, sourceDocument); - const root = rows.find((row) => row.title === FIXTURE_ROOT && !row.parent_source_section); - const children = rows - .filter((row) => root && row.parent_source_section === root.name) - .map((row) => row.title); - return ( - !!root && - rows.length === FIXTURE_SECTIONS.length + 1 && - JSON.stringify(children) === JSON.stringify(FIXTURE_SECTIONS) && - rows.every((row) => row.include_in_wiki === 1) +async function seedSource( + api: Api, + pdfMd5: string, + recorded?: ImportFixture, +): Promise { + const project = await findOrCreateProject(api, SOURCE_PROJECT_NAME); + const existing = await api.getList("Wikify Import", { + filters: { import_title: SOURCE_TITLE, project }, + fields: ["name", "status", "source_document", "pdf"], + }); + for (const imp of existing) { + let { status, source_document: sourceDocument } = imp; + const samePdf = (await api.getValue("File", { file_url: imp.pdf }, "content_hash")) === pdfMd5; + if (samePdf && IN_PROGRESS.includes(status)) { + const started = Date.now(); + ({ status, source_document: sourceDocument } = await waitForImport( + api, + imp.name, + ["Review", "Failed"], + SOURCE_PARSE_TIMEOUT, + )); + console.log(`[seed] waited ${seconds(started)} for the running parse ${imp.name}`); + } + if (samePdf && status === "Review" && sourceDocument) { + await renumberTree(api, sourceDocument); + const outline = await intactOutline( + api, + sourceDocument, + recorded?.import === imp.name ? recorded.outline : undefined, + ); + if (outline) return { import: imp.name, sourceDocument, title: SOURCE_TITLE, outline }; + } + await deleteImport(api, imp.name); + } + + const started = Date.now(); + const name = await startImport(api, { title: SOURCE_TITLE, project }); + const { source_document: sourceDocument } = await waitForImport( + api, + name, + "Review", + SOURCE_PARSE_TIMEOUT, ); + console.log(`[seed] parsed ${FIXTURE_PDF} as ${name} in ${seconds(started)}`); + const outline = await intactOutline(api, sourceDocument); + if (!outline) throw new Error(`${name} parsed into an unusable section tree`); + return { import: name, sourceDocument, title: SOURCE_TITLE, outline }; } -async function seedImport(api: Api, project: string, title: string): Promise { +async function seedClone( + api: Api, + project: string, + source: ImportFixture, + title: string, + recorded?: ImportFixture, +): Promise { const existing = await api.getList("Wikify Import", { filters: { import_title: title, project }, fields: ["name", "status", "source_document"], }); for (const imp of existing) { - if ( - imp.status === "Review" && - imp.source_document && - (await isPristine(api, imp.source_document)) - ) { - return { import: imp.name, sourceDocument: imp.source_document, title }; - } - if (IN_PROGRESS.includes(imp.status)) { - const done = await waitForImport(api, imp.name, ["Review", "Failed"]); - if (done.status === "Review" && (await isPristine(api, done.source_document))) { - return { import: imp.name, sourceDocument: done.source_document, title }; - } + if (imp.status === "Review" && imp.source_document && imp.name === recorded?.import) { + const outline = await intactOutline(api, imp.source_document, recorded.outline); + if (outline) + return { import: imp.name, sourceDocument: imp.source_document, title, outline }; } await deleteImport(api, imp.name); } - const name = await startImport(api, { title, project }); - const done = await waitForImport(api, name, "Review"); - return { import: name, sourceDocument: done.source_document, title }; + + const started = Date.now(); + const clone = await cloneImport(api, source.import, { title, project }); + console.log(`[seed] cloned ${source.import} into ${clone.import} in ${seconds(started)}`); + const outline = await intactOutline(api, clone.sourceDocument); + if (!outline || withoutNames(outline) !== withoutNames(source.outline)) { + throw new Error(`${clone.import} does not have the outline of ${source.import}`); + } + return { ...clone, title, outline }; +} + +function seconds(since: number): string { + return `${Math.round((Date.now() - since) / 1000)}s`; } setup("seed the fixture project and imports", async () => { - setup.setTimeout(2 * PARSE_TIMEOUT + 120_000); + setup.setTimeout(SOURCE_PARSE_TIMEOUT + 900_000); + const pdf = fs.readFileSync(FIXTURE_PDF); + const digest = createHash("sha256").update(pdf).digest("hex"); + if (FIXTURE_PDF_OVERRIDDEN) { + console.log(`[seed] E2E_FIXTURE_PDF is set: using ${FIXTURE_PDF}, sha256 check skipped`); + } else if (digest !== FIXTURE_PDF_SHA256) { + throw new Error(`${FIXTURE_PDF} has sha256 ${digest}, expected ${FIXTURE_PDF_SHA256}`); + } const api = await Api.create(); try { - const project = - (await findProject(api, PROJECT_NAME)) || (await createProject(api, PROJECT_NAME)); - // One at a time: parallel inserts on a fresh site deadlock creating the IMP- naming series row. - const review = await seedImport(api, project, `${FIXTURE_PREFIX} review`); - const agent = await seedImport(api, project, `${FIXTURE_PREFIX} agent`); - const fixtures: Fixtures = { project, projectName: PROJECT_NAME, review, agent }; + const previous = readPrevious(); + const source = await seedSource( + api, + createHash("md5").update(pdf).digest("hex"), + previous.source, + ); + const recorded = previous.source?.import === source.import ? previous : {}; + const project = await findOrCreateProject(api, PROJECT_NAME); + const review = await seedClone( + api, + project, + source, + `${FIXTURE_PREFIX} review`, + recorded.review, + ); + const agent = await seedClone(api, project, source, `${FIXTURE_PREFIX} agent`, recorded.agent); + const fixtures: Fixtures = { project, projectName: PROJECT_NAME, source, review, agent }; fs.mkdirSync(STATE_DIR, { recursive: true }); fs.writeFileSync(FIXTURES_FILE, JSON.stringify(fixtures, null, "\t")); } finally { diff --git a/frontend/e2e/specs/agent.spec.ts b/frontend/e2e/specs/agent.spec.ts index 6c66257..9eec3c9 100644 --- a/frontend/e2e/specs/agent.spec.ts +++ b/frontend/e2e/specs/agent.spec.ts @@ -2,14 +2,11 @@ import type { Locator, Page } from "@playwright/test"; import type { Api, Row } from "../helpers/api"; import { iconButton, waitForTurn } from "../helpers/assistant"; import { PREFIX } from "../helpers/env"; -import { expect, test } from "../helpers/test"; -import { rowOrder, sectionRow } from "../helpers/tree"; +import { expect, type FixtureOutline, type OutlineEntry, test } from "../helpers/test"; +import { openTree, rowOrder, sectionRow } from "../helpers/tree"; import { waitFor } from "../helpers/wait"; import { - FIXTURE_ROOT, - FIXTURE_SECTIONS, findFigure, - findSection, propagationPending, type SectionRow, sectionRows, @@ -24,10 +21,7 @@ const createdSessions = new Set(); type Turn = { session: Row; messages: Row[] }; async function openAssistant(page: Page, importName: string): Promise { - await page.goto(`/wikify/import/${importName}/tree`); - await expect( - page.getByRole("treeitem", { name: new RegExp(`^(Collapse |Expand )?${FIXTURE_ROOT}`) }), - ).toBeVisible(); + await openTree(page, importName); await iconButton(page, "lucide-sparkles").click(); const panel = page.getByRole("complementary").filter({ hasText: "Assistant" }); await expect(panel.getByPlaceholder("Ask the assistant…")).toBeVisible(); @@ -154,13 +148,77 @@ async function restoreContent( ); } +function treeShape(rows: SectionRow[]) { + return rows.map(({ name, title, parent_source_section }) => ({ + name, + title, + parent_source_section, + })); +} + +// A run of plain words that occurs once in the markdown, so a prompt can quote it and the render shows it as is. +function plainPhrase(markdown: string, words = 4): string | undefined { + for (const line of (markdown || "").split("\n")) { + if (/^\s*([#!|>]|[-*]\s|\d+\.)/.test(line)) continue; + const tokens = line.trim().split(/\s+/); + for (let start = 0; start + words <= tokens.length; start++) { + const run = tokens.slice(start, start + words); + if (!run.every((token) => /^[A-Za-z][A-Za-z'-]*$/.test(token))) continue; + const phrase = run.join(" "); + if (markdown.split(phrase).length === 2) return phrase; + } + } +} + +// A word in exactly one section body and in no title, so only a search tool can place it. +function bodyOnlyWord(sections: SectionRow[]): { word: string; owner: SectionRow } | undefined { + const bodies = sections.map((row) => + (row.markdown || "").replace(/!\[[^\]]*\]\([^)]*\)/g, "").replace(/\]\([^)]*\)/g, "]"), + ); + const titles = sections.map((row) => row.title.toLowerCase()).join("\n"); + const words = new Set( + bodies.flatMap((body) => body.match(/\b[A-Za-z][A-Za-z0-9]{5,}\b/g) || []), + ); + const rank = (word: string) => (/[a-z][A-Z]/.test(word) ? 0 : /^[A-Z]/.test(word) ? 1 : 2); + for (const word of [...words].sort((a, b) => rank(a) - rank(b))) { + const lower = word.toLowerCase(); + if (titles.includes(lower)) continue; + const owners = sections.filter((_, index) => bodies[index].toLowerCase().includes(lower)); + if (owners.length === 1) return { word, owner: owners[0] }; + } +} + +// A figure whose tag points at its page image and that exactly one section embeds. +async function pageImageFigure(api: Api, sourceDocument: string, sections: SectionRow[]) { + const pages = await api.getList("Source Page", { + filters: { source_document: sourceDocument }, + fields: ["name", "page_no", "image", "canonical_markdown"], + orderBy: "page_no asc", + }); + for (const page of pages) { + for (const [tag, caption, url] of (page.canonical_markdown || "").matchAll( + /!\[([^\]]*)\]\(([^)]*)\)/g, + )) { + if (!caption || caption.includes('"') || url !== page.image) continue; + const owners = sections.filter((row) => row.markdown?.includes(tag)); + if (owners.length === 1) + return { owner: owners[0], page: page.name, pageNo: page.page_no, caption, image: url }; + } + } +} + +function outlineRows(sections: SectionRow[], { root, children }: FixtureOutline): SectionRow[] { + const names = [...(root ? [root] : []), ...children].map((entry) => entry.name); + return sections.filter((row) => names.includes(row.name)); +} + async function expectPristine( api: Api, sourceDocument: string, sections: SectionRow[], ): Promise { const after = await sectionRows(api, sourceDocument); - expect(after.map((row) => row.title)).toEqual([FIXTURE_ROOT, ...FIXTURE_SECTIONS]); + expect(treeShape(after)).toEqual(treeShape(sections)); expect(after.every((row) => row.include_in_wiki === 1)).toBe(true); expect(after.map(({ name, markdown }) => ({ name, markdown }))).toEqual( sections.map(({ name, markdown }) => ({ name, markdown })), @@ -202,7 +260,10 @@ test.describe("assistant", () => { const treeCalls = toolRows(rows, ["read_tree"]); expect(treeCalls.length).toBeGreaterThan(0); const sections = await sectionRows(api, sourceDocument); - expect(sections.map((row) => row.title)).toEqual([FIXTURE_ROOT, ...FIXTURE_SECTIONS]); + const { root, children } = fixture.agent.outline; + expect(outlineRows(sections, fixture.agent.outline).map((row) => row.title)).toEqual( + [...(root ? [root] : []), ...children].map((entry) => entry.title), + ); // The stored tool result loses its
tags (Long Text is HTML-sanitised), so match titles and pages. const treeText = treeCalls.at(-1)!.content as string; for (const section of sections) { @@ -222,7 +283,10 @@ test.describe("assistant", () => { async ({ page, api, fixture }) => { test.setTimeout(TEST_TIMEOUT); const { sourceDocument } = fixture.agent; - const prompt = `${PREFIX} agent ${Date.now()} Use your search tool to find the sections of this document that mention PyMuPDF or OpenRouter, then list their titles.`; + const found = bodyOnlyWord(await sectionRows(api, sourceDocument)); + test.skip(!found, "no word of this parse sits in exactly one section body and no title"); + const { word, owner } = found!; + const prompt = `${PREFIX} agent ${Date.now()} Use your search tool to find the sections of this document that mention "${word}", then list their titles.`; const panel = await openAssistant(page, fixture.agent.import); await sendPrompt(panel, prompt); const sessionId = await findSession(api, sourceDocument, prompt); @@ -231,11 +295,10 @@ test.describe("assistant", () => { expect(finalReply(rows).status).toBe("done"); const searches = toolRows(rows, ["semantic_search", "search_sections"]); expect(searches.length).toBeGreaterThan(0); - const found = searches.map((row) => row.content).join("\n"); - // Both words appear only in section bodies, never in a title, so the tree alone cannot answer it. - for (const title of ["How Wikify processes a PDF", "Which AI service Wikify uses"]) { - expect(found, `search results name "${title}"`).toContain(title); - } + const results = searches.map((row) => row.content).join("\n"); + expect(results, `search results for "${word}" name "${owner.title}"`).toContain( + owner.title, + ); }, ); }); @@ -247,18 +310,20 @@ test.describe("assistant", () => { test.setTimeout(TEST_TIMEOUT); const { sourceDocument } = fixture.agent; const stamp = Date.now(); - const root = await findSection(api, sourceDocument, FIXTURE_ROOT); - const moved = await findSection(api, sourceDocument, "Finding content across documents"); - const renamed = await findSection(api, sourceDocument, "Reading the page review"); - const retyped = await findSection(api, sourceDocument, "The assistant"); - const movedIndex = FIXTURE_SECTIONS.indexOf(moved.title); + const { root, children } = fixture.agent.outline; + test.skip(!root, "this parse has no root section to move a child out of"); + const before = await sectionRows(api, sourceDocument); + const row = (entry: OutlineEntry) => before.find((section) => section.name === entry.name)!; + const movedIndex = 3; + const [renamed, retyped, moved] = [children[1], children[2], children[movedIndex]].map(row); const newTitle = `${PREFIX} agent renamed ${stamp}`; - const newType = - retyped.section_type === "training_and_education" - ? "quality_and_audits" - : "training_and_education"; + // Section types are per project, so pick another one this document already uses. + const newType = before + .map((section) => section.section_type) + .find((type) => type && type !== retyped.section_type); + test.skip(!newType, "this parse uses a single section type"); const prompt = - `${PREFIX} agent ${stamp} Do three things: 1) move the section '${moved.title}' (currently under '${FIXTURE_ROOT}') to the top level of the tree; ` + + `${PREFIX} agent ${stamp} Do three things: 1) move the section '${moved.title}' (currently under '${root!.title}') to the top level of the tree; ` + `2) rename the section '${renamed.title}' to '${newTitle}'; 3) set the section type of '${retyped.title}' to '${newType}'.`; let sessionId = ""; try { @@ -290,7 +355,7 @@ test.describe("assistant", () => { if (sessionId) await settle(api, sessionId); await api.call("wikify.api.sections.move_section", { name: moved.name, - new_parent: root.name, + new_parent: root!.name, new_index: movedIndex, }); await api.call("wikify.api.sections.rename_section", { @@ -302,10 +367,7 @@ test.describe("assistant", () => { section_type: retyped.section_type, }); } - expect((await sectionRows(api, sourceDocument)).map((row) => row.title)).toEqual([ - FIXTURE_ROOT, - ...FIXTURE_SECTIONS, - ]); + expect(treeShape(await sectionRows(api, sourceDocument))).toEqual(treeShape(before)); }, ); @@ -423,6 +485,7 @@ test.describe("assistant", () => { ).toEqual([true]); expect(await exists()).toBe(true); await expect(panel.getByText("Confirm delete_section")).toBeVisible(); + await expect(panel.getByText(`Delete section '${title}'.`, { exact: true })).toBeVisible(); await panel.getByRole("button", { name: "Cancel" }).click(); await expect(panel.getByText("Cancelled.")).toBeVisible(); expect(await exists()).toBe(true); @@ -449,9 +512,7 @@ test.describe("assistant", () => { expect(await exists()).toBe(false); await page.reload(); - await expect( - page.getByRole("treeitem", { name: new RegExp(`^(Collapse |Expand )?${FIXTURE_ROOT}`) }), - ).toBeVisible(); + await expect(page.locator("[data-section-row]").first()).toBeVisible(); await expect(page.getByText(title)).toHaveCount(0); } finally { if (sessionId) await settle(api, sessionId); @@ -468,21 +529,26 @@ test.describe("assistant", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/21" }, }, async ({ page, api, fixture }) => { - test.fail(); // known failure: #21. Remove test.fail() when the issue is closed. test.setTimeout(TEST_TIMEOUT * 2); const { sourceDocument } = fixture.agent; const stamp = Date.now(); const before = await sectionRows(api, sourceDocument); const pagesBefore = await pageSnapshot(api, sourceDocument); - const edited = before.find((row) => row.title === "How Wikify processes a PDF")!; - // Page 6 is shared by "The assistant" and "Which AI service Wikify uses". - const pageNo = 6; + const { children } = fixture.agent.outline; + const edited = before.find((row) => row.name === children[1].name)!; + const pageNo = children.at(-1)!.page_start; + const pageRow = pagesBefore.find((row) => row.page_no === pageNo)!; + const sectionPhrase = plainPhrase(edited.markdown); + const pagePhrase = plainPhrase(pageRow.canonical_markdown); + test.skip(!sectionPhrase || !pagePhrase, "no quotable phrase in the section or the page"); + const sectionEdit = `${sectionPhrase}, [test] section edit`; + const pageEdit = `${pagePhrase} ([test] page edit)`; const sectionPrompt = - `${PREFIX} agent ${stamp} In the section '${edited.title}', change the text 'and it costs nothing' to ` + - `'and it costs nothing, [test] section edit'. Change only that section.`; + `${PREFIX} agent ${stamp} In the section '${edited.title}', change the text '${sectionPhrase}' to ` + + `'${sectionEdit}'. Change only that section.`; const pagePrompt = - `Edit page ${pageNo} only (the page text, not any section): change 'Wikify calls one service' to ` + - `'Wikify calls one service ([test] page edit)'. Do not change any section.`; + `Edit page ${pageNo} only (the page text, not any section): change '${pagePhrase}' to ` + + `'${pageEdit}'. Do not change any section.`; let sessionId = ""; try { const panel = await openAssistant(page, fixture.agent.import); @@ -491,11 +557,11 @@ test.describe("assistant", () => { let rows = await waitForTurn(api, sessionId, sectionPrompt, TURN_TIMEOUT); expect(toolRows(rows, ["edit_section_content"]).length).toBeGreaterThan(0); expect(await api.getValue("Source Section", edited.name, "markdown")).toContain( - "and it costs nothing, [test] section edit", + sectionEdit, ); const article = page.getByRole("article"); await sectionRow(page, edited.name).getByText(edited.title, { exact: true }).click(); - await expect(article).toContainText("and it costs nothing, [test] section edit"); + await expect(article).toContainText(sectionEdit); // Sent before the reload: a reloaded page opens a new chat, not this session. await sendPrompt(panel, pagePrompt); @@ -507,18 +573,31 @@ test.describe("assistant", () => { await waitForPropagation(api, sourceDocument); await page.reload(); await sectionRow(page, edited.name).getByText(edited.title, { exact: true }).click(); - await expect(article).toContainText("and it costs nothing, [test] section edit"); - const pageRow = pagesBefore.find((row) => row.page_no === pageNo)!; + await expect(article).toContainText(sectionEdit); expect(await api.getValue("Source Page", pageRow.name, "canonical_markdown")).toContain( - "Wikify calls one service ([test] page edit)", + pageEdit, ); await page.goto(`/wikify/import/${fixture.agent.import}/pages?page=${pageNo}`); await expect(page.getByText("([test] page edit)")).toBeVisible(); + // A page edit reaches a section only when that section alone owns the page. + const ranges = await api.getList("Source Section", { + filters: { source_document: sourceDocument }, + fields: ["name", "page_start", "page_end", "lft", "rgt"], + }); + const covering = ranges.filter( + (row) => row.page_start <= pageNo && row.page_end >= pageNo, + ); + const owners = covering.filter( + (row) => !covering.some((other) => other.lft > row.lft && other.rgt < row.rgt), + ); + const changed = [edited.name, ...(owners.length === 1 ? [owners[0].name] : [])]; const after = await sectionRows(api, sourceDocument); + if (owners.length === 1) + expect(after.find((row) => row.name === owners[0].name)!.markdown).toContain(pageEdit); const others = (rows: SectionRow[]) => rows - .filter((row) => row.name !== edited.name) + .filter((row) => !changed.includes(row.name)) .map(({ title, markdown }) => ({ title, markdown })); expect(others(after)).toEqual(others(before)); } finally { @@ -540,7 +619,18 @@ test.describe("assistant", () => { const stamp = Date.now(); const before = await sectionRows(api, sourceDocument); const pagesBefore = await pageSnapshot(api, sourceDocument); - const target = before.find((row) => row.title === "Finding content across documents")!; + const candidates = fixture.agent.outline.children.flatMap((child) => { + const section = before.find((row) => row.name === child.name)!; + const sourcePage = pagesBefore.find((row) => row.page_no === section.page_start); + const phrase = plainPhrase(section.markdown); + return section.page_start === section.page_end && + phrase && + sourcePage?.canonical_markdown.includes(phrase) + ? [{ section, sourcePage, phrase }] + : []; + }); + test.skip(!candidates.length, "no one-page section quotes its page"); + const { section: target, sourcePage, phrase } = candidates[0]; const marker = `[test] drift marker ${stamp}`; const prompt = `${PREFIX} agent ${stamp} Rebuild the section '${target.title}' from its pages.`; let sessionId = ""; @@ -556,17 +646,15 @@ test.describe("assistant", () => { const rows = await waitForTurn(api, sessionId, prompt, TURN_TIMEOUT); const rebuilds = toolRows(rows, ["rebuild_section_from_pages"]); expect(rebuilds.map((row) => toolArgs(row).name)).toEqual([target.name]); - // Every fixture page is shared, so the rebuild adopts the whole page and the agent may then trim - // the neighbours' text off (the tool's overlap warning tells it to). The tool row proves the page was adopted. - const sourcePage = pagesBefore.find((row) => row.page_no === target.page_start)!; - expect(target.page_end).toBe(target.page_start); + // The rebuild adopts the whole page, and on a shared page the agent may then trim the + // neighbours' text off (the tool's overlap warning tells it to). The tool row proves the page was adopted. expect(rebuilds[0].content).toContain( `(${sourcePage.canonical_markdown.trim().length} chars)`, ); const after = await sectionRows(api, sourceDocument); const rebuilt = after.find((row) => row.name === target.name)!; expect(rebuilt.markdown).not.toContain(marker); - expect(rebuilt.markdown).toContain("The **Explore** view filters sections by label."); + expect(rebuilt.markdown).toContain(phrase); const others = (sections: SectionRow[]) => sections.filter((row) => row.name !== target.name); expect(others(after)).toEqual(others(drifted)); @@ -574,7 +662,7 @@ test.describe("assistant", () => { await page.reload(); await sectionRow(page, target.name).getByText(target.title, { exact: true }).click(); const article = page.getByRole("article"); - await expect(article).toContainText("The Explore view filters sections by label."); + await expect(article).toContainText(phrase); await expect(article).not.toContainText(marker); } finally { if (sessionId) await settle(api, sessionId); @@ -594,14 +682,10 @@ test.describe("assistant", () => { const stamp = Date.now(); const before = await sectionRows(api, sourceDocument); const pagesBefore = await pageSnapshot(api, sourceDocument); - const owner = before.find((row) => row.title === FIXTURE_SECTIONS[0])!; - const { pageNo, caption } = await findFigure(api, sourceDocument, { - fromPage: owner.page_start, - toPage: owner.page_end, - plainCaption: true, - }); + const found = await pageImageFigure(api, sourceDocument, before); + test.skip(!found, "no section embeds a page-image figure with a plain caption"); + const { owner, pageNo, caption, image } = found!; const target = pagesBefore.find((row) => row.page_no === pageNo)!; - const image: string = await api.getValue("Source Page", target.name, "image"); const placeholder = `/files/test-placeholder-${stamp}.png`; expect(target.canonical_markdown).toContain(`![${caption}](${image})`); expect(owner.markdown).toContain(`![${caption}](${image})`); @@ -660,12 +744,12 @@ test.describe("assistant", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/38" }, }, async ({ page, api, fixture }) => { - test.fail(); // known failure: #38. Remove test.fail() when the issue is closed. test.setTimeout(TEST_TIMEOUT); const { sourceDocument } = fixture.agent; const before = await sectionRows(api, sourceDocument); const pagesBefore = await pageSnapshot(api, sourceDocument); - const prompt = `${PREFIX} agent ${Date.now()} Crop the figure on page 1 (the screenshot of the Wikify Projects screen) to its top half.`; + const { pageNo, caption } = await findFigure(api, sourceDocument, { plainCaption: true }); + const prompt = `${PREFIX} agent ${Date.now()} Crop the figure "${caption}" on page ${pageNo} to its top half.`; let sessionId = ""; try { const panel = await openAssistant(page, fixture.agent.import); @@ -687,6 +771,7 @@ test.describe("assistant", () => { expect(await pageSnapshot(api, sourceDocument)).toEqual(pagesBefore); expect(await sectionRows(api, sourceDocument)).toEqual(before); // The wording is the bug (#38): the reply sends the user to the wiki editor, not the crop dialog. + test.fail(); // known failure: #38. Remove test.fail() when the issue is closed. expect(reply.content).toMatch(/crop/i); expect(reply.content).toMatch(/\bPages\b/); expect(reply.content).toMatch(/click/i); @@ -699,7 +784,7 @@ test.describe("assistant", () => { }, ); - // Every fixture page is shared by two sections, so the section side of the case is left to propagation (#21). + // The page is usually shared by two sections, so the section side of the case is left to propagation (#21). test( "F-AGENT-10 reparse one page with the VLM, with an instruction", { tag: ["@functional", "@llm", "@agent"] }, @@ -709,8 +794,18 @@ test.describe("assistant", () => { const stamp = Date.now(); const before = await sectionRows(api, sourceDocument); const pagesBefore = await pageSnapshot(api, sourceDocument); - const pageNo = 6; - const target = pagesBefore.find((row) => row.page_no === pageNo)!; + const markers = [ + { span: /`[^`\n]+`/g, name: "inline code span", mark: "backticks", word: /backtick/i }, + { span: /\*\*[^*\n]+\*\*/g, name: "bold span", mark: "asterisks", word: /asterisk/i }, + ]; + const marker = markers.find((entry) => + pagesBefore.some((row) => row.canonical_markdown?.match(entry.span)), + ); + test.skip(!marker, "no page of this parse has inline code or bold text"); + const { span, name: spanName, mark, word } = marker!; + const target = pagesBefore.find((row) => row.canonical_markdown?.match(span))!; + const pageNo: number = target.page_no; + const kept = plainPhrase(target.canonical_markdown.replace(span, ""), 2); const reparseFields = [ "canonical_source", "remediation_method", @@ -721,11 +816,9 @@ test.describe("assistant", () => { ]; const reparseBefore: Row = await api.getValue("Source Page", target.name, reparseFields); const meanBefore = await api.getValue("Source Document", sourceDocument, "canonical_mean"); - const backticked = /`[\w-]+\/[\w.-]+`/; - expect(target.canonical_markdown).toMatch(backticked); const prompt = `${PREFIX} agent ${stamp} Re-parse page ${pageNo} with the VLM, instruction: ` + - `"in the table, write every model name as plain text, without backticks".`; + `"write every ${spanName} as plain text, without ${mark}".`; let sessionId = ""; try { const panel = await openAssistant(page, fixture.agent.import); @@ -738,7 +831,7 @@ test.describe("assistant", () => { expect(reparses.length).toBeGreaterThan(0); const args = toolArgs(reparses.at(-1)!); expect(args).toMatchObject({ page_no: pageNo, method: "vlm" }); - expect(args.instruction).toMatch(/backtick/i); + expect(args.instruction).toMatch(word); expect(reparses.at(-1)!.content).toContain(`Re-parsed page ${pageNo} via vlm`); const reparsed: string = await api.getValue( @@ -747,13 +840,11 @@ test.describe("assistant", () => { "canonical_markdown", ); expect(reparsed).not.toBe(target.canonical_markdown); - expect(reparsed).toContain("OpenRouter"); - expect(reparsed).not.toMatch(backticked); + if (kept) expect(reparsed).toContain(kept); + expect(reparsed).not.toMatch(span); await page.goto(`/wikify/import/${fixture.agent.import}/pages?page=${pageNo}`); - await expect( - page.getByRole("cell", { name: "Re-read a page from its image" }), - ).toBeVisible(); + if (kept) await expect(page.getByText(kept).first()).toBeVisible(); } finally { if (sessionId) await settle(api, sessionId); await waitForPropagation(api, sourceDocument); @@ -773,15 +864,15 @@ test.describe("assistant", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/24" }, }, async ({ page, api, fixture }) => { - test.fail(); // known failure: #24. Remove test.fail() when the issue is closed. // Waits: queue (TURN_TIMEOUT) + stop (15 s) + follow-up or cleanup settle (TURN_TIMEOUT), so a timeout can't hide the bug. test.setTimeout(TURN_TIMEOUT * 2 + 120_000); const { sourceDocument } = fixture.agent; - const sectionCount = FIXTURE_SECTIONS.length + 1; + const sectionCount = (await sectionRows(api, sourceDocument)).length; + const pageCount = (await pageSnapshot(api, sourceDocument)).length; const prompt = `${PREFIX} agent ${Date.now()} Read the sections strictly one at a time: call read_section for ONE section, wait for its result, ` + `then call it for the next one. Never put more than one tool call in a step. Read all ${sectionCount} sections, ` + - `then call read_page for each of the 6 pages the same way, then list the sections without figures.`; + `then call read_page for each of the ${pageCount} pages the same way, then list the sections without figures.`; const followUp = "How many sections does this document have? Answer with one number."; let sessionId = ""; try { @@ -874,11 +965,11 @@ test.describe("assistant", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/25" }, }, async ({ page, api, fixture }) => { - test.fail(); // known failure: #25. Remove test.fail() when the issue is closed. test.setTimeout(TEST_TIMEOUT); const { sourceDocument } = fixture.agent; const stamp = Date.now(); const codeWord = `ZEPHYR${stamp % 100000}`; + const pageCount = (await pageSnapshot(api, sourceDocument)).length; const title = `${PREFIX} agent ${stamp} long history`; const { session_id: sessionId } = await api.call("wikify.api.agent.new_session", { scope: "document", @@ -891,7 +982,7 @@ test.describe("assistant", () => { const calls = Array.from({ length: 42 }, (_, index) => ({ id: `call_e2e_${stamp}_${index}`, name: "read_page", - args: { page_no: (index % 6) + 1 }, + args: { page_no: (index % pageCount) + 1 }, })); const history: Row[] = [ { @@ -907,7 +998,7 @@ test.describe("assistant", () => { tool_call_id: call.id, content: `Page ${call.args.page_no} read.`, })), - { role: "assistant", content: "I read all 6 pages seven times." }, + { role: "assistant", content: `I read all ${pageCount} pages seven times.` }, ]; for (const message of history) { await api.call("frappe.client.insert", { @@ -921,7 +1012,7 @@ test.describe("assistant", () => { const panel = await openAssistant(page, fixture.agent.import); await iconButton(page, "lucide-history", panel.locator("header")).click(); await page.getByRole("menuitem", { name: title }).click(); - await expect(panel.getByText("I read all 6 pages seven times.")).toBeVisible(); + await expect(panel.getByText(`I read all ${pageCount} pages seven times.`)).toBeVisible(); const prompt = "What was the code word I gave you in my first message of this chat? Reply with only the code word."; await sendPrompt(panel, prompt); diff --git a/frontend/e2e/specs/ask.spec.ts b/frontend/e2e/specs/ask.spec.ts index 7a3435d..c11b50d 100644 --- a/frontend/e2e/specs/ask.spec.ts +++ b/frontend/e2e/specs/ask.spec.ts @@ -90,7 +90,8 @@ test.describe("ask", () => { { tag: ["@functional", "@llm", "@ask"] }, async ({ page, api, fixture }) => { test.setTimeout(ANSWER_TIMEOUT + 60_000); - const question = `${AREA_PREFIX} Which AI service does Wikify use?`; + const topic = fixture.review.outline.children[1].title; + const question = `${AREA_PREFIX} What does the document say about "${topic}"?`; await page.goto("/wikify/ask"); await pickProject(page, fixture.projectName); @@ -142,10 +143,9 @@ test.describe("ask", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/53" }, }, async ({ page, api, fixture }) => { - // known failure: #53. Remove test.fail() when the issue is closed. - test.fail(); test.setTimeout(ANSWER_TIMEOUT + 60_000); - const question = `${AREA_PREFIX} What do the page review badges mean?`; + const topic = fixture.review.outline.children[2].title; + const question = `${AREA_PREFIX} What does the document say about "${topic}"?`; const countBefore = await api.call("frappe.client.get_count", { doctype: "Wikify Ask Session", filters: { project: fixture.project }, @@ -156,7 +156,7 @@ test.describe("ask", () => { await pickProject(page, fixture.projectName); await expect(page.getByPlaceholder("Ask a question of this wiki…")).toBeVisible(); // frappe-ui's Button overwrites aria-label with its (empty) label prop, so the icon-only - // history and trash buttons have no accessible name; find them by their icon. + // history button has no accessible name; find it by its icon. const history = page.getByRole("heading", { name: "History" }); if (!(await history.isVisible())) await page.locator("button:has(.lucide-history)").click(); @@ -171,9 +171,10 @@ test.describe("ask", () => { ).toBeVisible(); const row = page.getByRole("listitem").filter({ hasText: question }); - await row.locator("button:has(.lucide-trash-2)").click(); + await row.getByRole("button", { name: /^Delete conversation: / }).click(); const confirm = page.getByRole("dialog"); await expect(confirm).toBeVisible(); + await expect(confirm).toContainText(`Delete "${question}" and its answers?`); expect( await api.call("frappe.client.get_count", { doctype: "Wikify Ask Session", @@ -183,12 +184,15 @@ test.describe("ask", () => { await confirm.getByRole("button", { name: /Delete/ }).click(); await expect(row).toHaveCount(0); - expect( - await api.call("frappe.client.get_count", { - doctype: "Wikify Ask Session", - filters: { project: fixture.project }, - }), - ).toBe(countBefore); + // The row leaves the list before the delete request returns. + await expect + .poll(() => + api.call("frappe.client.get_count", { + doctype: "Wikify Ask Session", + filters: { project: fixture.project }, + }), + ) + .toBe(countBefore); expect( await api.call("frappe.client.get_count", { doctype: "Wikify Ask Message", diff --git a/frontend/e2e/specs/crop.spec.ts b/frontend/e2e/specs/crop.spec.ts index 91f2c13..60d6607 100644 --- a/frontend/e2e/specs/crop.spec.ts +++ b/frontend/e2e/specs/crop.spec.ts @@ -1,12 +1,7 @@ import type { Locator, Page, Request } from "@playwright/test"; import type { Api } from "../helpers/api"; import { expect, test } from "../helpers/test"; -import { - FIXTURE_SECTIONS, - findSection, - propagationPending, - setSectionMarkdown, -} from "../helpers/wikify"; +import { propagationPending, setSectionMarkdown } from "../helpers/wikify"; import { waitFor } from "../helpers/wait"; type PageSnapshot = { @@ -376,16 +371,30 @@ test.describe("crop", () => { async ({ page, api, fixture }) => { test.setTimeout(2 * PROPAGATION_TIMEOUT + 120_000); const sourceDocument = fixture.review.sourceDocument; - const section = await findSection(api, sourceDocument, FIXTURE_SECTIONS[0]); + // Only a page with one owner is propagated; a crop on a shared page edits the section holding the image. + const ownedFigurePages = (name: string) => + before.pages.filter((row) => { + const owners = owningSections(before.sections, row.page_no); + return ( + imageTags(row.canonical_markdown).length && + owners.length === 1 && + owners[0].name === name + ); + }); + const name = [ + ...fixture.review.outline.children.map((child) => child.name), + ...before.sections.map((row) => row.name), + ].find((candidate) => ownedFigurePages(candidate).length); + test.skip(!name, "no section of this parse alone owns a page with a figure"); + const section = { + name: name!, + title: await api.getValue("Source Section", name!, "title"), + }; const crops: { caption: string; url: string }[] = []; - const figurePages = before.pages.filter( - (row) => - row.page_no >= section.page_start && - row.page_no <= section.page_end && - imageTags(row.canonical_markdown).length, - ); - expect(figurePages.length, "figures on the section's pages").toBeGreaterThan(0); - for (const { page_no: pageNo, canonical_markdown } of figurePages) { + for (const { page_no: pageNo, canonical_markdown } of ownedFigurePages(section.name).slice( + 0, + 2, + )) { const [figure] = imageTags(canonical_markdown); const result = await api.call("wikify.api.pages.crop_page_figure", { source_document: sourceDocument, @@ -409,7 +418,7 @@ test.describe("crop", () => { (markdown) => crops.every((crop) => markdown.includes(crop.url)), { timeout: PROPAGATION_TIMEOUT, - label: "section markdown has both crops", + label: "section markdown has every crop", }, ); diff --git a/frontend/e2e/specs/parse.spec.ts b/frontend/e2e/specs/parse.spec.ts index ee8205b..a9d14f2 100644 --- a/frontend/e2e/specs/parse.spec.ts +++ b/frontend/e2e/specs/parse.spec.ts @@ -1,7 +1,7 @@ import type { Page } from "@playwright/test"; import type { Api } from "../helpers/api"; import { deleteTestProjects } from "../helpers/cleanup"; -import { FIXTURE_PDF, PREFIX } from "../helpers/env"; +import { PREFIX, WHAT_IS_WIKIFY_PDF } from "../helpers/env"; import { expect, test } from "../helpers/test"; import { waitFor, waitForValue } from "../helpers/wait"; import { @@ -21,11 +21,14 @@ const QUEUE_WAIT = 1_200_000; const REMEDIATE_TIMEOUT = 1_200_000; async function logEntries(api: Api, importName: string) { - return api.getList("Import Log Entry", { - filters: { import: importName }, - fields: ["name", "idx_seq", "stage", "message"], - orderBy: "idx_seq asc", - }); + return api.getList<{ name: string; idx_seq: number; stage: string; message: string }>( + "Import Log Entry", + { + filters: { import: importName }, + fields: ["name", "idx_seq", "stage", "message"], + orderBy: "idx_seq asc", + }, + ); } async function lastLogSeq(api: Api, importName: string): Promise { @@ -72,7 +75,6 @@ test.describe("parse", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/48" }, }, async ({ page, api }) => { - test.fail(); // known failure: #48. Remove test.fail() when the issue is closed. test.setTimeout(QUEUE_WAIT + 2 * PARSE_TIMEOUT); const projectName = `${AREA_PREFIX} watch`; const project = await createProject(api, projectName); @@ -83,7 +85,7 @@ test.describe("parse", () => { const dialog = page.getByRole("dialog"); const chooser = page.waitForEvent("filechooser"); await dialog.getByRole("button", { name: "Choose PDFs" }).click(); - await (await chooser).setFiles(FIXTURE_PDF); + await (await chooser).setFiles(WHAT_IS_WIKIFY_PDF); await dialog.getByLabel("Title").fill(title); await dialog.getByRole("button", { name: "Start", exact: true }).click(); @@ -142,7 +144,7 @@ test.describe("parse", () => { test.setTimeout(QUEUE_WAIT + 2 * PARSE_TIMEOUT); const project = await createProject(api, `${AREA_PREFIX} logs`); const title = `${AREA_PREFIX} logs`; - const importName = await startImport(api, { title, project }); + const importName = await startImport(api, { title, project, pdf: WHAT_IS_WIKIFY_PDF }); await page.goto(`/wikify/import/${importName}`); await page.getByRole("tab", { name: "Logs" }).click(); @@ -191,7 +193,7 @@ test.describe("parse", () => { test.setTimeout(QUEUE_WAIT + PARSE_TIMEOUT + 60_000); title = `${AREA_PREFIX} remediate`; const project = await createProject(api, title); - importName = await startImport(api, { title, project }); + importName = await startImport(api, { title, project, pdf: WHAT_IS_WIKIFY_PDF }); sourceDocument = (await waitForImport(api, importName, "Review", QUEUE_WAIT + PARSE_TIMEOUT)) .source_document; const parsedPages = await pageState(api, sourceDocument); @@ -202,7 +204,7 @@ test.describe("parse", () => { const seqBefore = await lastLogSeq(api, importName); await page.goto(`/wikify/import/${importName}`); await expect(page.getByRole("heading", { name: title })).toBeVisible(); - await page.getByRole("button", { name: "Remediate" }).click(); + await page.getByRole("button", { name: "Actions", exact: true }).click(); await page.getByRole("menuitem", { name: menuItem }).click(); await waitForValue(api, "Wikify Import", importName, "status", "Remediating", { timeout: 30_000, @@ -218,7 +220,6 @@ test.describe("parse", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/31" }, }, async ({ page, api }) => { - test.fail(); // known failure: #31. Remove test.fail() when the issue is closed. test.setTimeout(QUEUE_WAIT + REMEDIATE_TIMEOUT + 120_000); const parsed = await pageState(api, sourceDocument); // A clean parse flags no page, and then "Remediate flagged" has nothing to run on. @@ -259,6 +260,7 @@ test.describe("parse", () => { canonical_source, canonical_composite, })); + test.fail(); // known failure: #31. Remove test.fail() when the issue is closed. expect( canonical(await pageState(api, sourceDocument)), "passed pages keep their canonical read", @@ -321,20 +323,27 @@ test.describe("parse", () => { ], }, async ({ page, api }) => { - test.fail(); // known failure: #33, #50. Remove test.fail() when the issues are closed. test.setTimeout(QUEUE_WAIT + REMEDIATE_TIMEOUT); expect(await api.getValue("Wikify Import", importName, "status")).toBe("Review"); const seqBefore = await lastLogSeq(api, importName); await page.goto(`/wikify/import/${importName}`); await expect(page.getByRole("heading", { name: title })).toBeVisible(); - const remediate = page.getByRole("button", { name: "Remediate" }); - await expect(remediate).toBeVisible(); - const headerControl = page.getByRole("button", { name: /Reclassify/ }); - if (!(await headerControl.isVisible())) await remediate.click(); - const control = headerControl.or(page.getByRole("menuitem", { name: /Reclassify/ })); - await expect(control, "a Reclassify control on the import page").toBeVisible(); - await control.click(); + await page.getByRole("button", { name: "Actions", exact: true }).click(); + await page.getByRole("menuitem", { name: "Reclassify sections" }).click(); + const confirm = page.getByRole("dialog"); + await expect(confirm.getByText("Reclassify sections", { exact: true })).toBeVisible(); + await confirm.getByRole("button", { name: "Reclassify", exact: true }).click(); + await expect(confirm).toBeHidden(); + // The toast disappears after a few seconds, so watch for it while the job runs. + const toastShown = page + .getByRole("region", { name: /Notifications/ }) + .getByText("Sections reclassified") + .waitFor({ timeout: QUEUE_WAIT + REMEDIATE_TIMEOUT / 2 }) + .then( + () => true, + () => false, + ); const entries = await waitFor( async () => @@ -354,9 +363,7 @@ test.describe("parse", () => { for (const section of sections) expect(classified).toContain(`${section.title} → ${section.section_type}`); expect(await api.getValue("Wikify Import", importName, "status")).toBe("Review"); - await expect( - page.getByRole("region", { name: /Notifications/ }).getByText(/classif/i), - ).toBeVisible(); + expect(await toastShown, "a Sections reclassified toast").toBe(true); }, ); }); @@ -370,7 +377,7 @@ test.describe("parse", () => { async () => { test.fixme( true, - "needs a 400-page client PDF and over an hour of the only worker; the job is killed at its 3600 s RQ timeout (#22)", + "no public 400-page PDF in the suite; #22 (the 1 h job timeout) is fixed, so this needs only a large CC-licensed fixture", ); }, ); diff --git a/frontend/e2e/specs/publish.spec.ts b/frontend/e2e/specs/publish.spec.ts index fc98c32..8b4a1fa 100644 --- a/frontend/e2e/specs/publish.spec.ts +++ b/frontend/e2e/specs/publish.spec.ts @@ -5,12 +5,11 @@ import { PREFIX } from "../helpers/env"; import { expect, test } from "../helpers/test"; import { waitFor } from "../helpers/wait"; import { + cloneImport, createProject, - escapeRegExp, findFigure, outline, sectionRows, - startImport, waitForImport, type SectionRow, } from "../helpers/wikify"; @@ -52,6 +51,11 @@ async function expectRoutesOpen(page: Page, documents: WikiDocument[]): Promise< } } +async function firstLeaf(api: Api, sourceDocument: string): Promise { + const { rows, children } = await outline(api, sourceDocument); + return children.find((child) => !rows.some((row) => row.parent_source_section === child.name))!; +} + function ownParagraph(markdown: string): string { const paragraph = markdown .split(/\n\s*\n/) @@ -68,9 +72,10 @@ test.describe("publish", () => { const stamp = Date.now(); const name = `${PREFIX} publish ${stamp}`; const spaceRoute = `test-publish-${stamp}`; - const first = { import: "", sourceDocument: "" }; - const second = { import: "" }; + let first = { import: "", sourceDocument: "" }; + let second = { import: "", sourceDocument: "" }; let sections: SectionRow[] = []; + let root: SectionRow | undefined; let excluded: SectionRow; let leafTitle = ""; let otherTitle = ""; @@ -78,24 +83,32 @@ test.describe("publish", () => { let space = ""; let publishedDocuments: WikiDocument[] = []; - test.beforeAll(async ({ api }) => { + test.beforeAll(async ({ api, fixture }) => { test.setTimeout(QUEUE_TIMEOUT + 1_200_000); const project = await createProject(api, name); - first.import = await startImport(api, { title: name, project }); - second.import = await startImport(api, { title: `${name} 2`, project }); - first.sourceDocument = ( - await waitForImport(api, first.import, "Review", QUEUE_TIMEOUT) - ).source_document; + first = await cloneImport(api, fixture.source.import, { title: name, project }); + second = await cloneImport(api, fixture.source.import, { title: `${name} 2`, project }); - const { children } = await outline(api, first.sourceDocument); - excluded = children.at(-1)!; - leafTitle = children[0].title; - otherTitle = children[1].title; + let rows: SectionRow[]; + let children: SectionRow[]; + ({ rows, root, children } = await outline(api, first.sourceDocument)); + // A group's wiki route opens its first child, so the pages checked by title must be leaves. + const leaves = children.filter( + (child) => !rows.some((row) => row.parent_source_section === child.name), + ); + expect(leaves.length, "two top-level sections without children").toBeGreaterThanOrEqual(2); + const [leaf, other] = leaves; + excluded = children.findLast((child) => child !== leaf && child !== other)!; + leafTitle = leaf.title; + otherTitle = other.title; await api.call("wikify.api.sections.toggle_include", { name: excluded.name, include: 0 }); - // Page 3 on, not 1 or 2: a crop rebuilds every section covering the page from whole pages, and page 1 - // holds the root's own text, which would then leak into its first child and hide bug #30. - const figure = await findFigure(api, first.sourceDocument, { fromPage: 3 }); + // Past the first two children and both leaves: a crop rebuilds every section covering the page from + // whole pages, so the root's own text would leak into its first child (hiding bug #30), and a shared + // page would put one section's heading into another's page. + const figure = await findFigure(api, first.sourceDocument, { + fromPage: Math.max(children[1].page_end, leaf.page_end, other.page_end) + 1, + }); const result = await api.call("wikify.api.pages.crop_page_figure", { source_document: first.sourceDocument, page_no: figure.pageNo, @@ -137,15 +150,24 @@ test.describe("publish", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/44" }, }, async ({ page, api }) => { - // known failure: #44. Remove test.fail() when the issue is closed. - test.fail(); const preview = await api.call("wikify.api.imports.preview_wiki", { import_name: first.import, }); + const plural = (count: number, noun: string) => `${count} ${noun}${count === 1 ? "" : "s"}`; const dialog = await openPublishDialog(page, first.import); - await expect(dialog).toContainText( - new RegExp(`${escapeRegExp(leafTitle)}|\\b${preview.pages} pages?\\b`, "i"), - ); + await expect(dialog.getByText("Pages to publish", { exact: true })).toBeVisible(); + await expect( + dialog.getByText(`${plural(preview.pages, "page")} · ${plural(preview.groups, "group")}`, { + exact: true, + }), + ).toBeVisible(); + await expect( + dialog.getByText(`${preview.excluded} excluded`, { exact: true }), + ).toBeVisible(); + const tree = dialog.getByRole("tree"); + for (const node of preview.tree) + await expect(tree.getByText(node.title, { exact: true })).toBeVisible(); + await expect(tree.getByText(excluded.title, { exact: true })).toHaveCount(0); }, ); @@ -195,19 +217,13 @@ test.describe("publish", () => { annotation: { type: "issue", description: "https://github.com/bwhtech/wikify/issues/30" }, }, async ({ page }) => { - // known failure: #30. Remove test.fail() when the issue is closed. - test.fail(); - const root = sections.find( - (row) => - !row.parent_source_section && - sections.some((child) => child.parent_source_section === row.name), - ); test.skip(!root, "this parse produced no parent section"); - const parent = root!; + const parent = sections.find((row) => row.name === root!.name)!; const paragraph = ownParagraph(parent.markdown); expect(sections.filter((row) => row.markdown?.includes(paragraph))).toEqual([parent]); const group = publishedDocuments.find((document) => document.name === parent.wiki_document)!; await page.goto(`/${group.route}`); + test.fail(); // known failure: #30. Remove test.fail() when the issue is closed. await expect(page.getByRole("main")).toContainText(paragraph); }, ); @@ -260,12 +276,7 @@ test.describe("publish", () => { { tag: ["@functional", "@publish"] }, async ({ page, api }) => { test.setTimeout(QUEUE_TIMEOUT * 2); - const { source_document: sourceDocument } = await waitForImport( - api, - second.import, - "Review", - QUEUE_TIMEOUT, - ); + const { sourceDocument } = second; const before = await wikiDocuments(api, spaceRoute); await page.goto(`/wikify/import/${second.import}/tree`); @@ -306,9 +317,9 @@ test.describe("publish", () => { ); await expectRoutesOpen(page, added); - const secondLeaf = (await outline(api, sourceDocument)).children[0]; - const firstLeaf = (await outline(api, first.sourceDocument)).children[0]; - for (const leaf of [secondLeaf, firstLeaf]) { + const secondLeaf = await firstLeaf(api, sourceDocument); + const leafOfFirst = await firstLeaf(api, first.sourceDocument); + for (const leaf of [secondLeaf, leafOfFirst]) { const document = after.find((row) => row.name === leaf.wiki_document)!; await page.goto(`/${document.route}`); const main = page.getByRole("main"); @@ -334,7 +345,7 @@ test.describe("publish", () => { async () => { test.skip( true, - "BLOCKED: the dialog closes on Generate wiki and small imports generate in seconds; the only large import (R1) never reaches Review because its parse is killed at the 1 h RQ timeout.", + "generation makes no AI calls and commits every 50 pages, so even the 98-page fixture is published in seconds, too fast to stop from the UI", ); }, ); diff --git a/frontend/e2e/specs/smoke.spec.ts b/frontend/e2e/specs/smoke.spec.ts index 0232334..bf8f5cd 100644 --- a/frontend/e2e/specs/smoke.spec.ts +++ b/frontend/e2e/specs/smoke.spec.ts @@ -2,7 +2,8 @@ import type { Page } from "@playwright/test"; import type { Row } from "../helpers/api"; import { iconButton, waitForTurn } from "../helpers/assistant"; import { deleteTestProjects } from "../helpers/cleanup"; -import { env, FIXTURE_PDF, PREFIX } from "../helpers/env"; +import { env, PREFIX } from "../helpers/env"; +import { writePdfs } from "../helpers/pdf"; import { expect, test } from "../helpers/test"; import { childOrder, @@ -15,7 +16,15 @@ import { } from "../helpers/tree"; import { pickInDialog } from "../helpers/upload"; import { waitFor } from "../helpers/wait"; -import { findFigure, outline, pageRows, sectionRows, waitForImport } from "../helpers/wikify"; +import { + cloneImport, + createProject, + findFigure, + outline, + pageRows, + sectionRows, + waitForImport, +} from "../helpers/wikify"; const stamp = Date.now(); const NAME = `${PREFIX} smoke ${stamp}`; @@ -26,6 +35,7 @@ const IMAGE_TAG = /!\[([^\]]*)\]\(([^)]*)\)/g; const run = { project: "", + upload: "", import: "", sourceDocument: "", renamed: "", @@ -34,14 +44,13 @@ const run = { }; // The start of the first plain paragraph, cut before any inline markup so it matches the rendered text. -function plainSnippet(markdown: string): string { +function plainSnippet(markdown: string): string | undefined { for (const block of markdown.split(/\n\s*\n/)) { const text = block.trim().replace(/\s+/g, " "); if (/^(#|!\[|<|\||-|\*|>|\d)/.test(text)) continue; const plain = text.split(/[*_`[\]&<]/)[0].trim(); if (plain.length >= 20) return plain.slice(0, 60); } - throw new Error(`No plain paragraph in ${JSON.stringify(markdown.slice(0, 200))}`); } test.describe("smoke", () => { @@ -82,56 +91,64 @@ test.describe("smoke", () => { test( "S-02 upload a PDF into a new project and parse it", { tag: ["@smoke", "@upload"] }, - async ({ page, api }) => { + async ({ page, api }, testInfo) => { test.setTimeout(QUEUE_TIMEOUT + 300_000); + const uploadName = `${NAME} upload`; + const [pdf] = writePdfs(testInfo.outputPath("pdfs"), [uploadName]); await page.goto("/wikify/"); await page.getByRole("button", { name: "New Project" }).first().click(); const projectDialog = page.getByRole("dialog"); - await projectDialog.getByLabel("Project name").fill(NAME); + await projectDialog.getByLabel("Project name").fill(uploadName); await projectDialog.getByRole("button", { name: "Create" }).click(); await expect(page).toHaveURL(/\/wikify\/project\/PRJ-[\d-]+$/); - run.project = decodeURIComponent(page.url().split("/").pop()!); + const project = decodeURIComponent(page.url().split("/").pop()!); await page.getByRole("button", { name: "New Document" }).first().click(); const dialog = page.getByRole("dialog"); - await expect(dialog.getByLabel("Project")).toContainText(NAME); - await pickInDialog(page, FIXTURE_PDF); - await dialog.getByLabel("Title").fill(NAME); + await expect(dialog.getByLabel("Project")).toContainText(uploadName); + await pickInDialog(page, pdf); + await dialog.getByLabel("Title").fill(uploadName); await dialog.getByRole("button", { name: "Start", exact: true }).click(); await expect(page).toHaveURL(/\/wikify\/import\/IMP-[\d-]+$/); - run.import = decodeURIComponent(page.url().split("/").pop()!); - await expect(page.getByRole("heading", { name: NAME })).toBeVisible(); + run.upload = decodeURIComponent(page.url().split("/").pop()!); + await expect(page.getByRole("heading", { name: uploadName })).toBeVisible(); - const imp = await waitForImport(api, run.import, "Review", QUEUE_TIMEOUT); + const imp = await waitForImport(api, run.upload, "Review", QUEUE_TIMEOUT); expect(imp.error).toBeFalsy(); - expect(imp.stage_label).toBe("Parsed 6 pages"); - expect(await api.getValue("Wikify Import", run.import, ["project", "import_title"])).toEqual( - { - project: run.project, - import_title: NAME, - }, - ); - run.sourceDocument = imp.source_document; + expect( + await api.getValue("Wikify Import", run.upload, ["project", "import_title", "page_count"]), + ).toEqual({ project, import_title: uploadName, page_count: 1 }); + expect(await pageRows(api, imp.source_document)).toHaveLength(1); }, ); test( "S-03 Pages tab shows result markdown for every page", { tag: ["@smoke", "@tree"] }, - async ({ page, api }) => { + async ({ page, api, fixture }) => { + run.project = await createProject(api, NAME); + ({ import: run.import, sourceDocument: run.sourceDocument } = await cloneImport( + api, + fixture.source.import, + { title: NAME, project: run.project }, + )); const pages = await pageRows(api, run.sourceDocument); - expect(pages).toHaveLength(6); + expect(pages).toHaveLength(await api.getValue("Wikify Import", run.import, "page_count")); for (const row of pages) expect(row.canonical_markdown?.trim(), `page ${row.page_no}`).toBeTruthy(); const flagged = pages.filter((row) => row.verdict !== "pass").length; await page.goto(`/wikify/import/${run.import}/pages`); - await expect(page.getByRole("button", { name: "All (6)" })).toBeVisible(); + await expect(page.getByRole("button", { name: `All (${pages.length})` })).toBeVisible(); await expect(page.getByRole("button", { name: `Flagged (${flagged})` })).toBeVisible(); - await expect(page.getByRole("button", { name: `Passed (${6 - flagged})` })).toBeVisible(); + await expect( + page.getByRole("button", { name: `Passed (${pages.length - flagged})` }), + ).toBeVisible(); for (const row of pages) { + const snippet = plainSnippet(row.canonical_markdown); + if (!snippet) continue; await page.getByRole("button", { name: new RegExp(`^Page ${row.page_no}\\b`) }).click(); - await expect(page.getByText(plainSnippet(row.canonical_markdown))).toBeVisible(); + await expect(page.getByText(snippet)).toBeVisible(); } }, ); @@ -142,8 +159,12 @@ test.describe("smoke", () => { async ({ page, api }) => { const { rows: before, root, children: childRows } = await outline(api, run.sourceDocument); const children = childRows.map((row) => row.name); - const renamed = childRows[2]; - const excluded = childRows[3]; + // S-08 opens the renamed section's wiki page, and a group's route opens its first child instead. + const renamed = childRows + .slice(0, -1) + .find((child) => !before.some((row) => row.parent_source_section === child.name))!; + expect(renamed, "a top-level section without children").toBeTruthy(); + const excluded = [childRows[3], childRows[2]].find((row) => row.name !== renamed.name)!; const moved = children.at(-1)!; const newTitle = `${NAME} renamed`; @@ -173,7 +194,10 @@ test.describe("smoke", () => { sectionRow(page, excluded.name).getByText(excluded.title, { exact: true }), ).toHaveCSS("text-decoration-line", "line-through"); const heading = root ? [root.name] : []; - expect(await rowOrder(page)).toEqual([...heading, moved, ...children.slice(0, -1)]); + const shown = (await rowOrder(page)).filter( + (name) => name === root?.name || children.includes(name), + ); + expect(shown).toEqual([...heading, moved, ...children.slice(0, -1)]); const after = await sectionRows(api, run.sourceDocument); const byName = Object.fromEntries(after.map((row) => [row.name, row])); @@ -328,14 +352,15 @@ test.describe("smoke", () => { test( "S-09 Ask a question scoped to the project", { tag: ["@smoke", "@ask", "@llm"] }, - async ({ page, api }) => { + async ({ page, api, fixture }) => { test.setTimeout(QUEUE_TIMEOUT + 300_000); await waitFor( () => api.call("wikify.api.rag.index_status", { project: run.project }), (status) => status.documents === 1, { timeout: QUEUE_TIMEOUT, interval: 5_000, label: `${run.project} indexed` }, ); - const question = `${NAME} Which AI service does Wikify use, and where does its key live?`; + const topic = fixture.source.outline.children[1].title; + const question = `${NAME} What does the document say about "${topic}"?`; await page.goto("/wikify/ask"); await page.getByRole("combobox").click(); diff --git a/frontend/e2e/specs/tree.spec.ts b/frontend/e2e/specs/tree.spec.ts index 5b94326..1d17372 100644 --- a/frontend/e2e/specs/tree.spec.ts +++ b/frontend/e2e/specs/tree.spec.ts @@ -13,17 +13,16 @@ import { waitForSectionCall, } from "../helpers/tree"; import { + cloneImport, createProject, - FIXTURE_ROOT, - FIXTURE_SECTIONS, - findSection, outline, sectionRows, - startImport, waitForImport, type SectionRow, } from "../helpers/wikify"; +const IMAGE_TAG = /!\[[^\]]*\]\([^)]+\)/g; + async function expectStruck(page: Page, rows: SectionRow[], struck: boolean): Promise { for (const row of rows) { const title = sectionRow(page, row.name).getByText(row.title, { exact: true }); @@ -63,20 +62,17 @@ async function restoreTree( }); } } - const root = original.find((row) => !row.parent_source_section)!; - const childOrder = original - .filter((row) => row.parent_source_section === root.name) - .map((row) => row.name); - const currentOrder = (await sectionRows(api, sourceDocument)) - .filter((row) => row.parent_source_section === root.name) - .map((row) => row.name); - if (JSON.stringify(currentOrder) !== JSON.stringify(childOrder)) { - await api.call("wikify.api.sections.reorder_section", { - name: childOrder[0], - new_parent: root.name, - new_index: 0, - siblings: childOrder, - }); + const latest = await sectionRows(api, sourceDocument); + for (const parent of new Set(original.map((row) => row.parent_source_section || undefined))) { + const siblings = childOrder(original, parent); + if (JSON.stringify(childOrder(latest, parent)) !== JSON.stringify(siblings)) { + await api.call("wikify.api.sections.reorder_section", { + name: siblings[0], + new_parent: parent ?? "", + new_index: 0, + siblings, + }); + } } } @@ -95,7 +91,7 @@ test.describe("tree", () => { "F-TREE-01 rename persists after reload", { tag: ["@functional", "@tree", "@sanity"] }, async ({ page, api, fixture }) => { - const section = await findSection(api, fixture.review.sourceDocument, FIXTURE_SECTIONS[2]); + const section = fixture.review.outline.children[2]; const newTitle = `${PREFIX} tree ${Date.now()} renamed`; const renameCalls: string[] = []; page.on("request", (request) => { @@ -132,8 +128,9 @@ test.describe("tree", () => { "F-TREE-02 exclude a parent section strikes it and its children", { tag: ["@functional", "@tree", "@sanity"] }, async ({ page, api, fixture }) => { + const root = fixture.review.outline.root!; + test.skip(!root, "this parse has no root section"); const rows = await sectionRows(api, fixture.review.sourceDocument); - const root = rows.find((row) => row.title === FIXTURE_ROOT && !row.parent_source_section)!; const subtree = rows.filter( (row) => row.name === root.name || row.parent_source_section === root.name, ); @@ -159,8 +156,9 @@ test.describe("tree", () => { "F-TREE-03 include the parent section again", { tag: ["@functional", "@tree"] }, async ({ page, api, fixture }) => { + const root = fixture.review.outline.root!; + test.skip(!root, "this parse has no root section"); const rows = await sectionRows(api, fixture.review.sourceDocument); - const root = rows.find((row) => row.title === FIXTURE_ROOT && !row.parent_source_section)!; const subtree = rows.filter( (row) => row.name === root.name || row.parent_source_section === root.name, ); @@ -236,9 +234,12 @@ test.describe("tree", () => { { tag: ["@functional", "@tree"] }, async ({ page, api, fixture }) => { const sourceDocument = fixture.review.sourceDocument; - const root = original.find((row) => !row.parent_source_section)!; - const children = childOrder(original, root.name); + const root = fixture.review.outline.root?.name; + const children = childOrder(original, root); const moved = children.at(-1)!; + const heading = root ? [root] : []; + const shownOrder = async () => + (await rowOrder(page)).filter((name) => name === root || children.includes(name)); const others = children.slice(0, -1); let reorderCalls = 0; page.on("request", (request) => { @@ -256,9 +257,9 @@ test.describe("tree", () => { expect(reorderCalls).toBe(1); await page.reload(); await expect(sectionRow(page, moved)).toBeVisible(); - expect(await rowOrder(page)).toEqual([root.name, moved, ...others]); + expect(await shownOrder()).toEqual([...heading, moved, ...others]); let after = await sectionRows(api, sourceDocument); - expect(childOrder(after, root.name)).toEqual([moved, ...others]); + expect(childOrder(after, root)).toEqual([moved, ...others]); expect(after.find((row) => row.name === moved)!.sort_order).toBe(0); saved = waitForSectionCall(page, "reorder_section"); @@ -267,12 +268,11 @@ test.describe("tree", () => { expect(reorderCalls).toBe(2); await page.reload(); await expect(sectionRow(page, moved)).toBeVisible(); - expect(await rowOrder(page)).toEqual([root.name, ...others, moved]); + expect(await shownOrder()).toEqual([...heading, ...others, moved]); after = await sectionRows(api, sourceDocument); - expect(childOrder(after, root.name)).toEqual([...others, moved]); - expect(after.find((row) => row.name === moved)!.lft).toBe( - Math.max(...after.map((row) => row.lft)), - ); + expect(childOrder(after, root)).toEqual([...others, moved]); + const siblingLft = after.filter((row) => children.includes(row.name)).map((row) => row.lft); + expect(after.find((row) => row.name === moved)!.lft).toBe(Math.max(...siblingLft)); }, ); @@ -280,19 +280,33 @@ test.describe("tree", () => { "F-TREE-06 preview a section in the Wiki tab", { tag: ["@functional", "@tree"] }, async ({ page, api, fixture }) => { - const section = await findSection(api, fixture.review.sourceDocument, FIXTURE_SECTIONS[0]); + const names = fixture.review.outline.children.map((child) => child.name); + const rows = await sectionRows(api, fixture.review.sourceDocument); + const withFigure = rows.filter((row) => row.markdown?.match(IMAGE_TAG)); + // Top-level sections of a long chapter are often groups whose figures sit in their children. + const section = + withFigure.find((row) => names.includes(row.name)) ?? + withFigure.find((row) => row.parent_source_section)!; + expect(section, "a section with a figure").toBeTruthy(); + const ancestors: string[] = []; + for ( + let parent = section.parent_source_section; + parent; + parent = rows.find((row) => row.name === parent)!.parent_source_section + ) + ancestors.unshift(rows.find((row) => row.name === parent)!.title); + const imageCount = section.markdown.match(IMAGE_TAG)!.length; const documentTitle = await api.getValue( "Source Document", fixture.review.sourceDocument, "title", ); - expect(section.markdown.match(/!\[/g)).toHaveLength(2); await openTree(page, fixture.review.import, section.name); await sectionRow(page, section.name).getByText(section.title, { exact: true }).click(); await expect(page.getByRole("radio", { name: "Rendered" })).toBeChecked(); - const crumbs = [fixture.projectName, documentTitle, FIXTURE_ROOT, section.title]; + const crumbs = [fixture.projectName, documentTitle, ...ancestors, section.title]; await expect(page.getByRole("navigation").filter({ hasText: documentTitle })).toHaveText( new RegExp( `^${crumbs.map((crumb) => crumb.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")).join("\\s*›\\s*")}$`, @@ -302,7 +316,7 @@ test.describe("tree", () => { await expect(article.getByRole("heading", { level: 1, name: section.title })).toBeVisible(); await expect(article.locator("p").first()).not.toBeEmpty(); const images = article.locator("img"); - await expect(images).toHaveCount(2); + await expect(images).toHaveCount(imageCount); for (const image of await images.all()) { await image.scrollIntoViewIfNeeded(); await expect @@ -324,16 +338,13 @@ test.describe("tree after publish", () => { test( "F-TREE-07 edits after publish are rejected", { tag: ["@negative", "@tree"] }, - async ({ page, api }) => { + async ({ page, api, fixture }) => { test.setTimeout(2_700_000); const project = await createProject(api, name); - const importName = await startImport(api, { title: name, project }); - // One worker serves every agent's parses, so this parse can sit in Queued for a long while. - const { source_document: sourceDocument } = await waitForImport( + const { import: importName, sourceDocument } = await cloneImport( api, - importName, - "Review", - 1_800_000, + fixture.source.import, + { title: name, project }, ); const { rows: sections, root, children: childRows } = await outline(api, sourceDocument); const children = childRows.map((row) => row.name); @@ -346,7 +357,7 @@ test.describe("tree after publish", () => { import_name: importName, new_space: { space_name: name, route: `test-tree-${stamp}` }, }); - await waitForImport(api, importName, "Completed"); + await waitForImport(api, importName, "Completed", 1_800_000); const before = await sectionRows(api, sourceDocument); let reorderCalls = 0; diff --git a/wikify/agent_ci_probe.py b/wikify/agent_ci_probe.py new file mode 100644 index 0000000..05cb98b --- /dev/null +++ b/wikify/agent_ci_probe.py @@ -0,0 +1,80 @@ +import os +import shutil +import time + +import frappe + +from wikify.agent import session +from wikify.agent.loop import AgentRunner +from wikify.engine import claude_cli + + +def environment(): + print("which claude:", shutil.which("claude"), "| HOME:", os.environ.get("HOME")) + print("token set:", bool(os.environ.get("CLAUDE_CODE_OAUTH_TOKEN")), "| is_enabled:", claude_cli.is_enabled()) + try: + print("plain run:", claude_cli.run([{"role": "user", "content": "Reply with exactly: ok"}])["result"]) + except Exception as exception: + print("plain run FAILED:", repr(exception)) + + +def start_turn(): + sess = session.get_or_create(None, user="Administrator", scope="global") + session.append_message(sess.name, "user", "List my projects", status="done", attachments=[]) + session.set_running(sess.name, True) + frappe.db.commit() + return sess.name + + +def report(session_id, started): + print("turn took", round(time.time() - started, 1), "s") + for row in frappe.get_all( + "Wikify Agent Message", + filters={"session": session_id}, + fields=["role", "status", "content", "tool_name"], + order_by="creation asc", + ): + print(f" {row.role:9} {row.status or '':9} {row.tool_name or '':12} {(row.content or '')[:160]!r}") + for error in frappe.get_all("Error Log", fields=["method", "error"], order_by="creation desc", limit=5): + print("=== Error Log:", error.method) + print(error.error[-3000:]) + + +def direct(): + frappe.set_user("Administrator") + environment() + session_id = start_turn() + started = time.time() + AgentRunner(session_id, "Administrator", attachments=[]).run() + report(session_id, started) + + +def worker(): + frappe.set_user("Administrator") + frappe.db.delete("Error Log") + frappe.enqueue("wikify.agent_ci_probe.environment", queue="long") + session_id = start_turn() + started = time.time() + frappe.enqueue( + "wikify.jobs.agent.run_agent_job", queue="long", session_id=session_id, user="Administrator" + ) + frappe.db.commit() + while time.time() - started < 300: + time.sleep(5) + frappe.db.rollback() + if not frappe.db.get_value("Wikify Agent Session", session_id, "is_running"): + break + report(session_id, started) + + +def ask(): + frappe.set_user("Administrator") + from wikify.api.rag import ask as rag_ask + + try: + print("ask:", str(rag_ask("What is metabolism?"))[:300]) + except Exception as exception: + print("ask FAILED:", repr(exception)[:500]) + for error in frappe.get_all("Error Log", fields=["method", "error"], order_by="creation desc", limit=3): + print("=== Error Log:", error.method) + print(error.error[-2000:])