Repository navigation
fix(cli): scrape progress, working Back, multi-URL Enter; scrape folder as dataset input - #76
Merged
Merged
Conversation
…er as dataset input Scraping several URLs gave no feedback: the menu flow showed one spinner until every page was fetched, and `dataforge scrape` printed nothing until the end, so a rate-limited list looked stalled. scrape_urls now takes a progress callback (called before and after each URL); both entry points show a live bar naming the page being fetched and print each page's result as it finishes. With --json the lines go to stderr when it is a terminal. Back did not go back: questionary gives a Choice with value=None its title as the value, so Back in ask_input_method, ask_discovery_scope and ask_adjust_settings_target returned "(b) Back to menu" and similar instead of None. The wizard then re-showed the same step, and Back on the scope question saved "(b) Back" as discovery_scope. Back choices now use a sentinel mapped to None. Backing out of a URL box returns to the method list, Back on the method list returns to the main menu, and "back" on the first wizard step goes home instead of repeating it. The Multiple URLs box said "blank line to finish" but only finished on Alt+Enter, so Enter on a blank line looked unresponsive. Enter on an empty line now finishes, an empty box shows a validation message, and the number of URLs added is printed. New: Build an AI training dataset offers "Pages I scraped earlier". A folder from the no-AI scrape (pages.jsonl, or page_NNN.md when only Markdown was saved) becomes the session's collection via scrape.load_scrape_dir and scrape.import_into_session: raw files under sessions/<id>/raw plus DiscoveredURL and ScrapedPage rows, so resume and the later stages work unchanged. The run starts at processing; robots.txt and skip-known questions are skipped because nothing is fetched. Tests: key-press tests for Back and the URL box, wizard navigation, the progress callback order, loading a scrape folder both ways, and imported pages going through processing. README, CHANGELOG and TECHNICAL.tex updated.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three fixes for the interactive scrape and wizard flows, plus a new input option.
Fixed
No feedback while scraping several URLs
dataforge scrapeprinted nothing until the end. Pages on the same site are fetched one at a time to respect its rate limit, so a list could look stalled for a long time.scrape_urlstakes aprogresscallback (called before and after each URL). Both entry points show a live bar naming the page being fetched (Fetching 2/5) and print a line for each page as it finishes. With--json, the lines go to stderr, and only when stderr is a terminal, so stdout stays clean JSON.Back did not go back
Choicewithvalue=Noneits title as the value. Back inask_input_method,ask_discovery_scopeandask_adjust_settings_targetreturned strings like"(b) Back to menu"instead ofNone."(b) Back"asdiscovery_scope.None. Backing out of a URL box returns to the method list, Back on the method list returns to the main menu, and "back" on the first wizard step goes home instead of repeating it.Multiple URLs box looked frozen
Added
Pages I scraped earlier (in Build an AI training dataset)
./scrape/.scrape.load_scrape_dirreadspages.jsonl, or thepage_NNN.mdfiles when only Markdown was saved.scrape.import_into_sessionwrites the raw files undersessions/<id>/rawand addsDiscoveredURLandScrapedPagerows, so the later stages andresumework unchanged.Tests
tests/test_wizard_navigation.py: key-press tests for Back in each affected menu, the multi-URL box (finish on an empty line, empty Enter re-asks), backing out of a URL box, Back on the first wizard step, and choosing a scrape folder without being asked the scope.tests/test_quick_scrape.py: the order of progress callbacks, loading a scrape folder from JSONL and from Markdown only, and imported pages going throughProcessorAgent.Docs
README (scrape section), CHANGELOG (Unreleased) and TECHNICAL.tex (discovery section) updated.
Not in this PR
dataforge runrecipes and the MCPstart_runtool cannot take a scrape folder yet. Asource.scrape_dirrecipe field would cover both.