Skip to content

fix(cli): scrape progress, working Back, multi-URL Enter; scrape folder as dataset input - #76

Merged
ianktoo merged 1 commit into
masterfrom
fix/scrape-progress-back-import
Sep 25, 2026
Merged

ianktoo merged 1 commit into
masterfrom
fix/scrape-progress-back-import

Conversation

@ianktoo

@ianktoo ianktoo commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Summary

Three fixes for the interactive scrape and wizard flows, plus a new input option.

Fixed

No feedback while scraping several URLs

  • Before: Scrape pages (no AI) showed one spinner until every page was done, and dataforge scrape printed nothing until the end. Pages on the same site are fetched one at a time to respect its rate limit, so a list could look stalled for a long time.
  • Now: scrape_urls takes a progress callback (called before and after each URL). Both entry points show a live bar naming the page being fetched (Fetching 2/5) and print a line for each page as it finishes. With --json, the lines go to stderr, and only when stderr is a terminal, so stdout stays clean JSON.

Back did not go back

  • Cause: questionary gives a Choice with value=None its title as the value. Back in ask_input_method, ask_discovery_scope and ask_adjust_settings_target returned strings like "(b) Back to menu" instead of None.
  • Effect: the wizard re-showed the same step, and Back on the scope question saved "(b) Back" as discovery_scope.
  • Now: Back choices use a sentinel that is mapped to None. Backing out of a URL box returns to the method list, Back on the method list returns to the main menu, and "back" on the first wizard step goes home instead of repeating it.

Multiple URLs box looked frozen

  • Cause: the prompt said "blank line to finish", but questionary's multiline box only finishes on Alt+Enter, so Enter on a blank line just added another line.
  • Now: Enter on an empty line finishes. An empty box shows a validation message instead of submitting nothing, and the number of URLs added is printed.

Added

Pages I scraped earlier (in Build an AI training dataset)

  • Choose a folder written by the no-AI scrape. The default is the newest folder in ./scrape/.
  • scrape.load_scrape_dir reads pages.jsonl, or the page_NNN.md files when only Markdown was saved.
  • scrape.import_into_session writes the raw files under sessions/<id>/raw and adds DiscoveredURL and ScrapedPage rows, so the later stages and resume work unchanged.
  • The run starts at processing. The robots.txt and skip-known questions are skipped because nothing is fetched.
  • After a menu scrape, a hint points to this option.

Tests

  • New tests/test_wizard_navigation.py: key-press tests for Back in each affected menu, the multi-URL box (finish on an empty line, empty Enter re-asks), backing out of a URL box, Back on the first wizard step, and choosing a scrape folder without being asked the scope.
  • tests/test_quick_scrape.py: the order of progress callbacks, loading a scrape folder from JSONL and from Markdown only, and imported pages going through ProcessorAgent.
  • The full suite and ruff pass locally.

Docs

README (scrape section), CHANGELOG (Unreleased) and TECHNICAL.tex (discovery section) updated.

Not in this PR

dataforge run recipes and the MCP start_run tool cannot take a scrape folder yet. A source.scrape_dir recipe field would cover both.

…er as dataset input

Scraping several URLs gave no feedback: the menu flow showed one spinner
until every page was fetched, and `dataforge scrape` printed nothing until
the end, so a rate-limited list looked stalled. scrape_urls now takes a
progress callback (called before and after each URL); both entry points
show a live bar naming the page being fetched and print each page's result
as it finishes. With --json the lines go to stderr when it is a terminal.

Back did not go back: questionary gives a Choice with value=None its title
as the value, so Back in ask_input_method, ask_discovery_scope and
ask_adjust_settings_target returned "(b) Back to menu" and similar instead
of None. The wizard then re-showed the same step, and Back on the scope
question saved "(b) Back" as discovery_scope. Back choices now use a
sentinel mapped to None. Backing out of a URL box returns to the method
list, Back on the method list returns to the main menu, and "back" on the
first wizard step goes home instead of repeating it.

The Multiple URLs box said "blank line to finish" but only finished on
Alt+Enter, so Enter on a blank line looked unresponsive. Enter on an empty
line now finishes, an empty box shows a validation message, and the number
of URLs added is printed.

New: Build an AI training dataset offers "Pages I scraped earlier". A folder
from the no-AI scrape (pages.jsonl, or page_NNN.md when only Markdown was
saved) becomes the session's collection via scrape.load_scrape_dir and
scrape.import_into_session: raw files under sessions/<id>/raw plus
DiscoveredURL and ScrapedPage rows, so resume and the later stages work
unchanged. The run starts at processing; robots.txt and skip-known
questions are skipped because nothing is fetched.

Tests: key-press tests for Back and the URL box, wizard navigation, the
progress callback order, loading a scrape folder both ways, and imported
pages going through processing. README, CHANGELOG and TECHNICAL.tex updated.
@ianktoo
ianktoo merged commit f2a7b0c into master Sep 25, 2026
20 checks passed
@ianktoo ianktoo mentioned this pull request Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant