Add king-scrape update for incremental corpus refresh (ADR-0014)

[nelsonmfinda]

Summary

king-scrape update <name>. Refetches upstream of an indexed corpus, rechunks, reuses every section whose chunked content is byte identical to the previous scrape. Only new or changed chunks hit the LLM, so a typical refresh costs cents instead of dollars. The integrity unit is the chunk's content_hash from ADR-0012; reused sections carry forward keywords, use_cases, tags, priority but adopt fresh title, path, url so an upstream page reorganisation is reflected.

Related issue

Closes #18

Type of change

• New feature (non-breaking change that adds functionality)

How it was tested

pytest -q

725 passed, 1 skipped

24 tests in tests/test_scraper/test_update.py:

• Pure unit logic: _section_hash (with and without _meta), _resolve_source_url (meta vs base_url fallback), _build_reuse_index, _plan_update classification + added/removed URL
• Pure unit logic: _section_hash (with and without _meta), _resolve_source_url (meta vs base_url fallback), _build_reuse_index, _plan_update classification + added/removed URL diffs, _materialise_reused (carries enrichment, takes fresh identity), _interleave_in_chunk_order ordering
• Corpus loading: missing corpus, malformed JSON
• End-to-end with mocked pipeline: reuse + enrich path produces correct merged JSON, refuse-to-wipe-empty guard, short-circuit when all reused, lost-chunk count surfaced when enrichment validation fails
• Work-dir reset: stale pages/, chunks/, enriched/, manifest.json all wiped; safe on clean dir
• Atomic write: complete file, no leftover tempfile
• CLI: --corpus-path bypasses lookup, missing path returns 1, --provider flag overrides env, user-abort returns 1, missing corpus returns 1

Haven't run a live update against a real corpus in this PR. Happy to do that as part of review if you want; the audit subcommand from #N (PR 2) plus this one is the natural pair to demo on data/elevenlabs-api.json.

Checklist

• My code follows the project style (English-only code, comments, and identifiers)
• I ran the test suite locally and it passes (pytest)
• I added or updated tests where it made sense
• I updated the documentation in docs/ and/or README.md if behavior changed (new "Refresh an indexed corpus" section in docs/CLI_GUIDE.md, command overview row, ADR-0014 added under .king-context/adr/, CHANGELOG entry)
• I read the Contributing guide and the Code of Conduct
• My commits follow the project commit message style
• I confirmed there are no secrets or credentials in the diff

Additional notes

A few choices worth flagging:

• Subcommand dispatch follows the same pattern as audit from Add king-scrape audit subcommand for corpus drift #49 (PR 2): a four-line if argv[1] == "update": branch in cli.main. No argparse subparser refactor. Two subcommands now use this dispatch; a third would justify the refactor.
• Work directory is reset at the start of every update (pages/, chunks/, enriched/, manifest.json wiped). Without this, two distinct critical bugs fire: stale enriched/batch_*.json triggers the resume logic in enrich.py and silently returns OLD enriched chunks instead of fresh ones; and orphan pages/*.md from URLs no longer accepted by the
  filter survive chunk_pages and resurface as "reused" sections in the new corpus. Both bugs surfaced in the senior code review pass before push and are now covered by tests. At 10x corpora the reset matters more, not less; cheap and unconditional is the right call.
• Refuse to wipe a non-empty corpus to empty. If discover or filter produces zero URLs (network blip, over-aggressive regex), the update aborts with ValueError before writing and the original data/<name>.json is untouched. Tested.
• Atomic write back via tempfile + os.replace. An interrupted update never leaves a partial JSON file in data/<name>.json.
• fetch_pages gains a force_refresh=True keyword arg. Update calls it with True so changed upstream pages are actually re-downloaded instead of being skipped by the existing slug-presence resume logic. Provider Protocols (ADR-0010) are untouched: this is a pipeline-side flag.
• Short-circuit when nothing changed. If every fresh chunk hashes to an existing section, enrich_chunks is not called at all (saves one disk-stat round trip and is unambiguously correct). Tested.
• Lost chunks are visible. If 3 of 50 new chunks fail validation through the LLM retries, UpdateReport.lost = 3, a warning prints, and the CLI summary includes lost 3. Pre-fix this was a silent drop.
• --corpus-path flag for non-standard layouts (CI artifacts, monorepo subprojects). Bypasses the default data/<name>.json and .king-context/data/<name>.json lookup.
• --provider flag overrides env. Existing cli.run_pipeline uses setdefault and silently ignores the flag when SCRAPE_PROVIDER is already set; new code in update fixes that
  footgun for its own surface (os.environ["SCRAPE_PROVIDER"] = ...). Stage-specific envs (SCRAPE_DISCOVER_PROVIDER, SCRAPE_FETCH_PROVIDER) still take precedence per ADR-0009. The same fix should land in cli.run_pipeline as a follow-up, but that's out of scope here.
• auto_seed=False on the export. Update writes back to the corpus committed in the repo, not to the local indexed MCP store. After a successful update, refresh the local index with kctx index .king-context/data/<name>.json (or kctx index --all). Matches the project's existing ethos: indexed retrieval state is reproducible from the JSON committed in the repo, never the source of truth.

Independent of audit functionally; the two compose well as a pair (audit to see drift, update to act on it). ADR-0014 records the rationale and rejects the alternatives (page-level reuse, full rescrape under a flag, diff-against-audit-report, mutate corpus in place without atomic write).

[deandevz]

Hey Nelson, took a careful read through. Strong PR, the audit + update
pair closes the loop on #18 cleanly and the ADR-0012 chunk identity is
the right unit to hang the reuse logic on.

What stood out as good:

• Reused sections carry forward enrichment but adopt fresh title, path,
  url. Two changed paragraphs in a 50 chunk page only re enrich those
  two chunks, and a page reorganisation upstream is reflected without a
  re enrich pass. The _materialise_reused test pins this behaviour
  directly. Exactly the trade off I wanted.
• The _reset_work_dir docstring spells out why the wipe is
  unconditional (stale enriched/batch_*.json triggers the resume in
  enrich.py and silently returns OLD batches; orphan pages/*.md
  resurface as reused). Real bugs, fix in the right place, covered by
  tests. Cheap and unconditional is the right call.
• Refuse to wipe a non empty corpus to empty. The test asserts the
  on-disk JSON is byte identical after the ValueError, not just that
  the function raised. That is the level of assertion I want to see on
  data preserving guards.
• Atomic write via tempfile + os.replace with cleanup on every failure
  path, plus a test that asserts no .tmp leak. Same shape as the
  enrichment cache write from feat(scraper): content hash provenance and enrichment cache #46, consistent.
• Lost chunks are visible (UpdateReport.lost, warning, summary line).
  Pre fix this would have been a silent drop on validation failure.
• --provider overrides env. Direct os.environ[\"SCRAPE_PROVIDER\"] = ... here instead of the setdefault footgun in cli.run_pipeline.
  ADR-0009 stage specific envs still take precedence, with a test.
• ADR-0014 lists real alternatives (page level reuse, full rescrape,
  diff against audit, in place mutation) with grounded rejections. Same
  level as the ADR-0012 writeup. Useful as a reviewer artifact.

What I need before I merge:

• One end to end run against a live corpus, like we did for feat(scraper): content hash provenance and enrichment cache #46. Happy
  to run it on my side and post the results back here so you do not
  have to spin up the providers again, just say the word and I will
  take it. If you want to do it yourself, that works too. Either way
  the shape that would close it for me is the same: run audit first,
  then update on the same corpus, and confirm three things on disk.
  (1) a second update with no upstream changes shows reused N, enriched 0 and git diff on data/<name>.json is empty or only
  _meta.scraped_at. (2) a forced edit on one upstream page (or one
  staged content tweak) shows that exact section as the only re
  enriched one. (3) lost 0 in the summary. Hono is fine, or
  data/elevenlabs-api.json for the natural pair with audit.

A couple of things I noticed but I am leaving to your judgment. If
you agree they are worth addressing, go ahead. If you look at them
and decide they are fine as is, that works for me too:

1. ADR-0014 vs code drift. The `Alternatives Considered` section
   still says atomic write via tempfile + replace is a future
   hardening, but `_atomic_write_json` already implements it in this
   PR. Tiny edit on that paragraph would keep the ADR honest.

2. Pre ADR-0012 corpora fall back to `base_url` for the source URL.
   In legacy corpora `base_url` is often the host root, not the
   original entry point of the scrape, so a refresh could redis-
   cover a wider URL set than the original. Worth a one line warning
   when the fallback fires, or a note in the docs that legacy corpora
   should be re scraped from scratch first. Not blocking.

3. The cost confirmation prompt fires even when every fresh chunk
   was reused (cost is $0, but the user still gets asked). Short
   circuiting on `not plan.new_chunks` would skip a redundant prompt.
   Pure UX nit.

On the `setdefault` fix you flagged in the description, splitting
that into a follow up issue against `cli.run_pipeline` is the right
call. Nothing in this PR depends on it.

Let me know how you want to play the e2e and I will move on the merge
once it is green.

[deandevz]

Ran the e2e you proposed in my earlier review against data/minimax-audio.json (legacy corpus, 12 sections curated, $0.06 of OpenRouter spend total). Six findings, one of them blocking. The core flow works: reuse by content_hash hit 296 of 297 chunks on a no change rerun, staged drift on one corrupted hash triggered exactly one re enrich, lost 0 held. The blocker is the legacy base_url fallback path, which wiped the curated 12 to 360 different sections and spent OpenRouter credits without warning. Details and repro in #55.

I consolidated feedback across your four open PRs (#51, #52, #53, #54) into that issue and suggested unifying them into one PR rather than juggling four with overlapping cli.py edits. Open to whichever shape works on your side. Happy to run the e2e again on whatever you ship.

[deandevz]

Superseded by #56, merged as ae1f941. The update flow,
ADR-0014, and the six safety fixes from the e2e review all
landed verbatim in the unified stack. Thanks for the
original work, closing as superseded.