fix(remediate): save pages in batches and resume a killed import

[Rl0007]

1. A long PDF killed by its background job during remediation lost every page it had remediated, and the import stayed on Remediating with no way to continue.
2. Remediation now saves the canonical markdown every 10 pages. An import left in Parsing or Remediating with no job behind it shows "Resume import" in the Actions menu, which continues from the first unsaved page without parsing the PDF again.

You can visit an import's page in /wikify after a killed run and open Actions to see it.

Fixes #22

#86 merged into #85's branch after #85 had merged, so this ports it to main.

Before

After

Verification

• Ported onto current main. Main already queues parse and remediation under one job id per import (wikify_import:<name>), so resume uses that id instead of the separate parse/remediate ids from fix(remediate): save canonical pages in batches and resume a killed run #86.
• test_killed_run_keeps_saved_batches_and_resume_skips_them fails on main (0 != 1) and passes here; the 4 TestResumeImport tests error on main (5 errors) and pass here.
• Full suite on the devbox: 558 tests, 0 failures, 11 skipped (main: 553, 0 failures, 11 skipped; the skips are the corpus-only RAG eval tests).
• Devbox end to end: a 6-page [test] import with a batch size of 2 had its job process SIGKILLed at page 5 (pages 1-4 saved, status stayed Remediating). "Resume import" showed in Actions; clicking it ran pages 5-6 only on the same Source Document and the page moved to Review live. The page model, cleanup and judge calls were stubbed (the devbox has no model key).
• Reviews: Vue approved. Frappe review agreed with the job-id resolution and raised two open questions: main's stalled-import check marks a stuck import Failed after 10 minutes, which hides Resume (the Failed path re-runs Remediate from scratch); and resuming a killed manual re-remediation skips every page, because each page already has a canonical source from the first run.

[vishwajeet-13]

I checked out this PR and tried it on my local bench. I used a 25-page PDF and fake AI calls (same as your devbox test), then killed the job on purpose.

What works: Kill the job at page 18 → pages 1-10 are saved → "Resume import" shows up → Resume only does pages 11-25 (15 pages) and ends in Review. 👍

I found 4 cases where it breaks. One comment each below.

[vishwajeet-13]

Problem 1: The Resume button goes away after about 10 minutes.

There is a cron job (wikify/jobs/stalled.py) that runs every 10 minutes. It sees the import has no job, and marks it Failed.
After that, "Resume import" is gone. Only "Remediate flagged" / "Remediate all pages" are left.

I clicked "Remediate all pages" after this. It did all 25 pages again. The 10 saved pages were wasted.

So if nobody opens the page in those 10 minutes, this PR does not help at all.

Idea: also allow Resume when the status is Failed, if the import has a source_document and some pages are not saved yet.

[vishwajeet-13]

Problem 2: If the job times out, there is no Resume.

The job timeout (6 hours) throws JobTimeoutException. In RQ this is a normal Exception, so the except Exception in parse.run catches it and sets the status to Failed.

I made the job time out at page 18. Pages 1-10 were saved, but the import went to Failed and there was no "Resume import".

A very long PDF hitting the timeout is the main reason a run gets killed. Today Resume only shows up when the process is hard-killed (out of memory, worker restart).

[vishwajeet-13]

Problem 3: Resume after a killed "Remediate all pages" does nothing, and says Done.

Steps:

1. Import a PDF fully. It reaches Review.
2. Click "Remediate all pages" (or ask the agent to re-parse with an instruction).
3. The job gets killed after page 4. The status stays Remediating, so "Resume import" shows up.
4. Click Resume.

What happens: Resume runs parse.run, not remediate.run. Every page already has a canonical_source from the first import, so every page is skipped (0 AI calls). The log says "Remediating all 25 pages" and then straight away "Done … status Review". Pages 5-25 were never redone. The agent's instruction and the scope are lost too.

Idea: save which job was running (and its scope/instruction) on the import, and resume that same job. Or don't show Resume for a killed re-remediation.

[vishwajeet-13]

Problem 4: Killed while parsing → Resume makes a second Source Document.

parse_pdf creates the Source Document at the start and saves pages one by one. But imp.source_document is only set after parsing ends.

I killed the job at page 12 of parsing (status Parsing). Then I clicked Resume. It parsed the whole PDF again into a new Source Document. The old one, with 11 pages, is still there and still points to the same import.

Idea: set source_document on the import right after it is created, or delete the half-done one before parsing again.