From a2eb3747e58093e7451a80eba4b751fbd15bf3a7 Mon Sep 17 00:00:00 2001 From: "Ian T." Date: Thu, 24 Sep 2026 15:48:19 -0700 Subject: [PATCH 1/3] chore: ignore the local .dataforge session index The .dataforge file lists local sessions with absolute database and output paths from the user's machine, so it should not be committed. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 8a1bc96..52f4a22 100644 --- a/.gitignore +++ b/.gitignore @@ -89,6 +89,7 @@ Thumbs.db # Project-specific dataforge.db +.dataforge run.bat run.ps1 run.sh From 37fc86b67ee17012ea8b2c18c354f33f6a4bbe3c Mon Sep 17 00:00:00 2001 From: "Ian T." Date: Thu, 24 Sep 2026 16:43:49 -0700 Subject: [PATCH 2/3] feat(cli): keyboard URL review, single-page scope, language choice; fix global API key URL review (cli/url_review.py) is now a full-screen keyboard list: arrow keys move, Space ticks or unticks, Left and Right turn pages (the pager shows arrows where there are more URLs), a ticks or unticks everything shown, / filters as you type, i shows details, Enter finishes, Esc goes back. The page fits the terminal. The typed commands still work after ':'; row numbers now refer to the whole list. On a site with pages in several languages (/es/..., ?lang=fr), the review first asks which language to tick; the others stay listed, unticked, and l cycles the list through one language at a time. Language detection moves from app.py to url_review.py and reuses the recipe's locale list. Discovery scope: for seed URLs that are not a sitemap, the wizard asks whether to scrape only those pages (discovery and review skipped), follow their links (crawl from the seeds without the sitemap), or take the whole site. PipelineContext.discovery_scope carries the choice; it is saved in the session config so resume repeats it. Page scope ignores skip_known. Fix: a key saved with "Save API key globally" lives in the OS keychain, but startup read only prefs.json and only when the folder had no .env, which `dataforge config` always creates, so runs still asked for the key. preflight.apply_saved_config now loads saved keys, provider and model with the precedence shell env, then .env, then global config; the credentials check looks in the keychain before prompting; and a provider changed from the menu updates the loaded settings. Tests: key-press tests for the review screen, scope tests for the explorer, and saved-key precedence tests. TECHNICAL.tex discovery section updated. --- CHANGELOG.md | 28 ++ docs/TECHNICAL.tex | 13 +- src/dataforge/agents/base.py | 4 + src/dataforge/agents/explorer.py | 14 +- src/dataforge/agents/orchestrator.py | 4 +- src/dataforge/cli/app.py | 68 ++- src/dataforge/cli/preflight.py | 51 +- src/dataforge/cli/prompts.py | 30 ++ src/dataforge/cli/ui.py | 6 + src/dataforge/cli/url_review.py | 705 ++++++++++++++++++++------- tests/test_autosave.py | 2 +- tests/test_quality_and_lang.py | 2 +- tests/test_scope_and_saved_keys.py | 139 ++++++ tests/test_url_review.py | 160 +++--- 14 files changed, 930 insertions(+), 296 deletions(-) create mode 100644 tests/test_scope_and_saved_keys.py diff --git a/CHANGELOG.md b/CHANGELOG.md index bad0cff..6aa8624 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,34 @@ Format loosely follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Added +- **Scrape just the page you gave**: after entering URLs that are not a + sitemap, the wizard asks whether to scrape only those pages (no discovery, + no URL review), follow the links from them (a crawl from the pages, without + the sitemap), or take the whole site as before. Resume repeats the same + choice. +- **Language choice in the URL review**: on a site with pages in several + languages (`/es/...`, `?lang=fr`), the review first asks which language to + tick; the others stay listed, unticked. `l` cycles the list through one + language at a time. + +### Changed +- **Keyboard URL review**: arrow keys move, Space ticks or unticks, Left and + Right turn pages (the pager shows arrows where there are more URLs), `a` + ticks or unticks everything shown, `/` filters as you type, `i` shows + details, Enter finishes and Esc goes back. The page fits the terminal. The + typed commands (`x 3-8`, `f blog`, ...) still work after `:`; row numbers + now refer to the whole list, not only the current page. + +### Fixed +- **Globally saved API key not found**: "Save API key globally" stores the + key in the OS keychain, but startup only read `prefs.json`, and only when + the folder had no `.env` (which `dataforge config` always creates), so the + run still asked for the key. Saved keys, provider and model are now loaded + whether or not `.env` exists (shell environment first, then `.env`, then the + global config), the key check looks in the keychain before prompting, and a + provider changed from the menu takes effect without a restart. + ## [2.6.0] - 2026-09-24 A simpler interactive menu with a fixed screen layout, help at every stage and diff --git a/docs/TECHNICAL.tex b/docs/TECHNICAL.tex index e8ecf3b..ce71e22 100644 --- a/docs/TECHNICAL.tex +++ b/docs/TECHNICAL.tex @@ -201,10 +201,15 @@ \subsection{Discovery} (\texttt{robots.txt}, rate limit, DataForge's User-Agent), does not load images, media or fonts, and is not used at all on a site that declares a \texttt{Crawl-delay}, because the sub-requests of a page load cannot be -spaced one per delay. In interactive use, discovery is followed by a -review step --- substring, glob, or regex filtering plus per-URL -selection --- that a recipe replaces declaratively with -\texttt{source.include} / \texttt{source.exclude} / \texttt{source.max\_urls}. +spaced one per delay. In interactive use, seed URLs that are not a +sitemap come with a choice of scope: only the given pages (discovery is +skipped), the pages they link to (a crawl from the seeds without the +sitemap), or the whole site. Discovery is then followed by a review +step: on a site served in several languages the user first picks the one +to keep, then filters by substring, glob or regular expression and ticks +URLs from the keyboard. A recipe replaces this step declaratively with +\texttt{source.language} / \texttt{source.include} / +\texttt{source.exclude} / \texttt{source.max\_urls}. \subsection{Collection} \label{sec:collection} diff --git a/src/dataforge/agents/base.py b/src/dataforge/agents/base.py index 99da62c..fdae920 100644 --- a/src/dataforge/agents/base.py +++ b/src/dataforge/agents/base.py @@ -40,6 +40,10 @@ class PipelineContext: n_per_chunk: int = 3 ignore_robots: bool = False skip_known: bool = False # skip URLs already scraped in prior sessions + # How far discovery reaches from the seed URLs: "site" reads the site's + # sitemap (crawling when there is none), "links" crawls from the seeds + # without the sitemap, "page" uses the seeds as given. + discovery_scope: str = "site" # The recipe's language/include/exclude check. The fallback crawl spends # its page budget only on URLs that pass it (#63). None = keep all. url_filter: Callable[[str], bool] | None = None diff --git a/src/dataforge/agents/explorer.py b/src/dataforge/agents/explorer.py index 5682c5d..c766d69 100644 --- a/src/dataforge/agents/explorer.py +++ b/src/dataforge/agents/explorer.py @@ -49,8 +49,9 @@ async def run(self) -> PipelineContext: # Deduplicate (already maintained by source_map insertion order) discovered = all_urls - # Optionally skip URLs already scraped in any previous session - if self.ctx.skip_known and discovered: + # Optionally skip URLs already scraped in any previous session. Pages + # named one by one are what the user asked for, so they are kept. + if self.ctx.skip_known and discovered and self.ctx.discovery_scope != "page": discovered = self._filter_already_scraped(discovered) self.ctx.discovered_urls = discovered @@ -70,8 +71,12 @@ async def _explore_seed(self, client, seed: str) -> tuple[list[str], str]: urls = await parse_sitemap(client, seed) return (urls if urls else [seed], URLSource.sitemap) + scope = self.ctx.discovery_scope + if scope == "page": + return ([seed], URLSource.manual) + # 2. Try to discover sitemaps (every one robots.txt lists, #60) - sitemap_urls = await discover_sitemap_urls(client, base) + sitemap_urls = [] if scope == "links" else await discover_sitemap_urls(client, base) sitemap_url = ", ".join(sitemap_urls) if sitemap_urls: raw_urls = await parse_sitemaps(client, sitemap_urls) @@ -104,7 +109,8 @@ async def _explore_seed(self, client, seed: str) -> tuple[list[str], str]: # 3. Fall back to BFS crawl from seed URL self.log.info( - f"No usable sitemap for {base}, starting BFS crawl " + f"{'Following links from ' + seed if scope == 'links' else 'No usable sitemap for ' + base}, " + "starting BFS crawl " f"(max_pages={self.ctx.settings.max_crawl_pages}, max_depth={self.ctx.settings.max_crawl_depth})" ) crawled = await crawl( diff --git a/src/dataforge/agents/orchestrator.py b/src/dataforge/agents/orchestrator.py index 0afc0ae..ce5b85a 100644 --- a/src/dataforge/agents/orchestrator.py +++ b/src/dataforge/agents/orchestrator.py @@ -212,7 +212,9 @@ def _init_session(self) -> None: seed_urls=json.dumps(self.ctx.seed_urls), # A recipe may set its own output_dir; later commands # (stats, resume) need it to find this session's files. - config_json=json.dumps({"output_dir": str(s.output_dir.resolve())}), + # discovery_scope lets resume repeat discovery the same way. + config_json=json.dumps({"output_dir": str(s.output_dir.resolve()), + "discovery_scope": self.ctx.discovery_scope}), )) db.commit() diff --git a/src/dataforge/cli/app.py b/src/dataforge/cli/app.py index eaed43c..a093ede 100644 --- a/src/dataforge/cli/app.py +++ b/src/dataforge/cli/app.py @@ -426,6 +426,7 @@ def _label(x: PipelineSession) -> str: format=DataFormat(session.format), seed_urls=session.seed_url_list(), settings=s, + discovery_scope=session.config().get("discovery_scope", "site"), ) # Re-hydrate prior-stage data from DB so _checkpoint() doesn't zero out saved counts @@ -804,10 +805,16 @@ async def _configure() -> None: from dataforge.cli import prefs as user_prefs user_prefs.set("llm_provider", provider) user_prefs.set("llm_model", model) + # Settings is loaded once per process: without this, a change made from + # the menu only took effect after a restart, and the next run still + # checked the old provider's key. + s = get_settings() + s.llm_provider = provider + s.llm_model = model if info.requires_key: import getpass - existing = os.getenv(info.key_env, "") + existing = os.getenv(info.key_env, "") or user_prefs.get_api_key(info.key_env) masked = f"{existing[:8]}..." if len(existing) > 8 else ("set" if existing else "") prompt_label = ( f" {info.key_env} [{masked}] (leave blank to keep): " @@ -825,7 +832,9 @@ async def _configure() -> None: updated = _set_env_var(updated, info.key_env, key_value) env_path.write_text("\n".join(updated) + "\n") ui.success(f"Saved {info.key_env} to {env_path.resolve()}") - elif not existing: + elif existing: + os.environ.setdefault(info.key_env, existing) + else: ui.info(f"No key entered — set {info.key_env} via 'dataforge config' when ready") ui.success(f"Saved: provider={provider}, model={model}") @@ -1463,7 +1472,7 @@ async def _browse_discovered_urls(session_id: str) -> None: ui.info("No URLs discovered yet for this session.") return ui.info(f"[dim]Browsing {len(urls)} discovered URLs — changes are not saved[/]") - await run_url_review(urls) # read-only: return value discarded + await run_url_review(urls, ask_language=False) # read-only: return value discarded # ── Explore menu (interactive data browser) ──────────────────────────────────── @@ -1582,6 +1591,15 @@ async def _step_urls(state: dict) -> StepResult: ui.error("No valid URLs provided") return "back" state["seed_urls"] = result + # A sitemap already lists the pages to choose from; anything else could + # mean one page or a whole site, so ask rather than always crawling. + if all(u.lower().endswith(".xml") for u in result): + state["discovery_scope"] = "site" + else: + scope = await prompts.ask_discovery_scope(len(result)) + if scope is None: + return "back" + state["discovery_scope"] = scope return "next" @@ -1640,8 +1658,11 @@ async def _step_config(state: dict) -> StepResult: from urllib.parse import urlparse as _urlparse seed_domain = _urlparse(state.get("seed_urls", [""])[0]).netloc or "this site" - skip_known = await prompts.ask_skip_known(seed_domain) - state["skip_known"] = skip_known + # Pages named one by one are scraped even if seen before. + if state.get("discovery_scope", "site") != "page": + state["skip_known"] = await prompts.ask_skip_known(seed_domain) + else: + state["skip_known"] = False threshold = await prompts.ask_quality_threshold() if threshold is None: @@ -1776,6 +1797,7 @@ async def _interactive_pipeline() -> None: n_per_chunk=state["n_per_chunk"], ignore_robots=state.get("ignore_robots", False), skip_known=state.get("skip_known", False), + discovery_scope=state.get("discovery_scope", "site"), quality_threshold=state.get("quality_threshold", 0.5), generation_model=state.get("generation_model", ""), quality_model=state.get("quality_model", ""), @@ -1887,23 +1909,6 @@ async def _run_wizard(state: dict) -> str: PipelineStage.quality: PipelineStage.export, } -_LANG_RE = re.compile( - r"/([a-z]{2}(?:-[a-z]{2})?)/|[?&]lang(?:uage)?=([a-z]{2})|[?&]locale=([a-z]{2})", - re.IGNORECASE, -) - - -def _detect_language_groups(urls: list[str]) -> dict[str, int]: - """Return {locale_code: count} for URLs containing language/locale patterns.""" - counts: dict[str, int] = {} - for url in urls: - m = _LANG_RE.search(url) - if m: - lang = next(g for g in m.groups() if g).lower() - counts[lang] = counts.get(lang, 0) + 1 - return counts - - async def _adjust_settings(context: PipelineContext) -> None: """Mid-session settings menu — change the model or output dir between stages. @@ -2006,19 +2011,12 @@ async def post_discovery_hook(stage: str, context: PipelineContext) -> bool: ui.warn("Discovery returned 0 URLs. Check that the site has a reachable sitemap or provide a direct sitemap URL.") return False - # The review screen lists the URLs itself; printing them here as well - # only pushed the review's key hints off screen. - notice = "" - lang_groups = _detect_language_groups(context.discovered_urls) - if len(lang_groups) >= 2: - top = ", ".join(f"{k} ({v})" for k, v in sorted(lang_groups.items(), key=lambda x: -x[1])[:5]) - notice = ( - f"Several languages found: {top}. " - "To keep one, filter it, e.g. [bold]f /en/[/], then [bold]none[/] / [bold]all[/] as needed." - ) - - from .url_review import run_url_review - selected = await run_url_review(context.discovered_urls, notice=notice) + if context.discovery_scope == "page": + # The user named the pages; there is nothing to choose from. + selected = list(context.discovered_urls) + else: + from .url_review import run_url_review + selected = await run_url_review(context.discovered_urls) if not selected: ui.warn("No URLs selected — returning to menu.") diff --git a/src/dataforge/cli/preflight.py b/src/dataforge/cli/preflight.py index f9bed68..40a03c2 100644 --- a/src/dataforge/cli/preflight.py +++ b/src/dataforge/cli/preflight.py @@ -34,28 +34,48 @@ } +def apply_saved_config(env_path: Path | None = None) -> None: + """Load the global config into the process environment. + + Precedence: the process environment, then the project's .env, then the + global config saved by ``dataforge config``. Keys saved globally live in + the OS keychain (prefs.json only when no keychain is available), so both + are read. A key found in .env is exported too: litellm reads keys from the + environment, not from Settings. + """ + from dotenv import dotenv_values + + from dataforge.cli import prefs as user_prefs + + env_path = env_path or Path(".env") + in_file = {k: v for k, v in dotenv_values(env_path).items() if v} if env_path.exists() else {} + + for var, pref in (("DATAFORGE_LLM_PROVIDER", "llm_provider"), ("DATAFORGE_LLM_MODEL", "llm_model")): + saved = user_prefs.get(pref) + if saved and not os.getenv(var) and var not in in_file: + os.environ[var] = saved + + key_envs = {k for k in _PROVIDER_KEY_MAP.values() if k} + key_envs.update(user_prefs.load().get("api_keys", {})) + for key_env in sorted(key_envs): + if os.getenv(key_env): + continue + value = in_file.get(key_env) or user_prefs.get_api_key(key_env) + if value: + os.environ[key_env] = value + + def check_env_file() -> bool: """Warn (not fatal) if .env doesn't exist or no provider key is configured.""" from dataforge.cli import prefs as user_prefs env_path = Path(".env").resolve() - env_exists = env_path.exists() - if not env_exists: + if not env_path.exists(): show_warning( f".env file not found at {env_path}", f"Run: dataforge config to save settings globally to {user_prefs._prefs_path()}", ) - # Apply saved prefs as env var defaults so pip-installed users keep their config - saved_provider = user_prefs.get("llm_provider") - saved_model = user_prefs.get("llm_model") - if saved_provider and not os.getenv("DATAFORGE_LLM_PROVIDER"): - os.environ["DATAFORGE_LLM_PROVIDER"] = saved_provider - if saved_model and not os.getenv("DATAFORGE_LLM_MODEL"): - os.environ["DATAFORGE_LLM_MODEL"] = saved_model - # Apply globally-saved API keys so pip-installed users work without .env - for key_env, value in user_prefs.load().get("api_keys", {}).items(): - if value and not os.getenv(key_env): - os.environ[key_env] = value + apply_saved_config(env_path) # Check whether the active provider is satisfied (keyless like Ollama, or has a key set) s = get_settings() @@ -85,6 +105,11 @@ def check_llm_credentials() -> tuple[bool, str | None]: return _check_openai_compatible(s.local_base_url, s.local_api_key) if key_env and not os.getenv(key_env) and not getattr(s, key_env.lower(), ""): + from dataforge.cli import prefs as user_prefs + saved = user_prefs.get_api_key(key_env) + if saved: + os.environ[key_env] = saved + return True, None # Offer a live prompt before failing try: value = getpass.getpass(f" {key_env} not set — paste your key now (hidden, session-only): ") diff --git a/src/dataforge/cli/prompts.py b/src/dataforge/cli/prompts.py index c33cac6..7b57024 100644 --- a/src/dataforge/cli/prompts.py +++ b/src/dataforge/cli/prompts.py @@ -120,6 +120,36 @@ async def ask_input_method() -> str | None: return answer +async def ask_discovery_scope(n_urls: int) -> str | None: + """How far to go from the URLs given: just them, their links, or the whole site.""" + one = n_urls == 1 + return await questionary.select( + "Scrape just this page, or search for more?" if one + else "Scrape just these pages, or search for more?", + choices=[ + questionary.Choice( + "Just this page" if one else "Just these pages", value="page", + description="No discovery: go straight to scraping " + f"{'this URL' if one else 'these URLs'}.", + ), + questionary.Choice( + "Deep link search: follow links from " + ("this page" if one else "these pages"), + value="links", + description="Crawl the pages they link to on the same site, " + "then pick which to keep.", + ), + questionary.Choice( + "The whole site", value="site", + description="Read the site's sitemap (or crawl it when there is none), " + "then pick which pages to keep.", + ), + questionary.Separator(), + questionary.Choice("(b) Back", value=None), + ], + **_q(), + ).ask_async() + + async def ask_single_url() -> str | None: url = await questionary.text( "Enter URL:", diff --git a/src/dataforge/cli/ui.py b/src/dataforge/cli/ui.py index d0eb623..980e175 100644 --- a/src/dataforge/cli/ui.py +++ b/src/dataforge/cli/ui.py @@ -157,6 +157,12 @@ def review_panel(state: dict) -> None: t.add_row("", url) if len(seed_urls) > 5: t.add_row("", f"[dim]... and {len(seed_urls) - 5} more[/]") + scope = { + "page": "Only the URL(s) above", + "links": "The URL(s) above and the pages they link to", + "site": "The whole site", + }.get(state.get("discovery_scope", "site"), "") + t.add_row("Scrape", scope) t.add_row("Session", state.get("session_name", "")) goal_text = state.get("goal", "")[:80] t.add_row("Goal", goal_text) diff --git a/src/dataforge/cli/url_review.py b/src/dataforge/cli/url_review.py index f3b9cd3..45a493e 100644 --- a/src/dataforge/cli/url_review.py +++ b/src/dataforge/cli/url_review.py @@ -1,76 +1,161 @@ -"""Interactive URL review step — paginated browser with filter, select, and inspect.""" +"""Interactive URL review step: a keyboard-driven list with filter, language and inspect. + +Arrow keys move through the list, Space ticks or unticks a page, Left and Right +turn pages, ``l`` switches between the site's languages, ``/`` filters as you +type. The typed commands of earlier versions (``x 3-8``, ``f blog``, ...) are +still available after ``:``. +""" from __future__ import annotations import re -from urllib.parse import urlparse +from urllib.parse import parse_qs, urlparse import questionary -from prompt_toolkit import PromptSession +from prompt_toolkit.application import Application, get_app +from prompt_toolkit.buffer import Buffer +from prompt_toolkit.filters import Condition +from prompt_toolkit.key_binding import KeyBindings +from prompt_toolkit.layout import ( + ConditionalContainer, + HSplit, + Layout, + VSplit, + Window, +) +from prompt_toolkit.layout.controls import BufferControl, FormattedTextControl from prompt_toolkit.styles import Style as PTStyle from questionary import Style as QStyle -from rich import box -from rich.table import Table +from dataforge.cli.recipe import url_locale from dataforge.collectors import filter_urls from . import ui _PAGE_SIZE = 30 - -_HELP_TEXT = ( - "Type a command, then Enter:\n" - " [bold]done[/] scrape the ticked pages [bold]f blog[/] show only URLs containing 'blog'\n" - " [bold]x 3[/] untick row 3 ([bold]x 3-8[/] for a range) [bold]+ 3[/] tick it again " - "[bold]all[/] / [bold]none[/]\n" - " [bold]n[/] / [bold]p[/] next / previous page [bold]i 3[/] details of row 3 " - "[bold]back[/] main menu [bold]?[/] full help" +# Lines the screen uses besides the URL rows: title, rule, column header, +# pager, status, two hint lines, input line and some slack. +_CHROME_LINES = 10 + +_HINTS = ( + "↑↓ move space tick/untick ←→ page a tick/untick all shown " + "l language / filter\n" + "i details : type a command ? help enter done esc back" ) _HELP_FULL = """\ -URL Review — full command reference -──────────────────────────────────────────────────────────────────────────────── - n / next Go to next page - p / prev Go to previous page - Go to page number - f Filter URLs (substring, /path/* glob, re:) - f Clear current filter — show all URLs - x <#> Deselect URL by row number on current page - x <#>-<#> Deselect a range of rows (e.g. x 3-8) - + <#> Select (re-include) URL by row number - + <#>-<#> Select a range of rows - all Select all URLs in the current filtered view - none Deselect all URLs in the current filtered view - i <#> Inspect URL — show full details for that row - done Confirm selection and proceed to collection - back / b / q / quit Cancel — go back to the main menu - ? Show this help -────────────────────────────────────────────────────────────────────────────────""" +Keys + ↑ ↓ / j k Move up and down + space Tick or untick the page under the cursor + ← → / PgUp PgDn Previous / next page of the list + Home / End First / last URL + a Tick all shown URLs (untick them if all are ticked) + l Show one language at a time (cycles through them) + / Filter as you type (substring, /path/* glob, re:) + i Details of the URL under the cursor + enter Done: scrape the ticked pages + esc / q Back to the main menu + +Commands (press : first) + x 3 / x 3-8 Untick row 3 / rows 3 to 8 + + 3 / + 3-8 Tick them again + all / none Tick / untick every shown URL + f / f Filter / clear the filter + Go to that page + done / back Same as enter / esc""" _PT_STYLE = PTStyle.from_dict({ - "prompt": "bold cyan", - "auto-suggestion": "fg:ansibrightblack italic", + "title": "bold ansicyan", + "subtitle": "ansibrightblack", + "rule": "ansicyan", + "header": "bold", + "rownum": "ansibrightblack", + "tick": "ansigreen bold", + "untick": "ansibrightblack", + "cursor": "reverse", + "notice": "ansiyellow", + "message": "ansicyan", + "pager": "ansicyan", + "pager.off": "ansibrightblack", + "hint": "ansibrightblack", + "detail": "", + "prompt": "bold ansicyan", }) _RANGE_RE = re.compile(r"^(\d+)-(\d+)$") +_LANG_NAMES = { + "ar": "Arabic", "bn": "Bengali", "de": "German", "el": "Greek", "en": "English", + "es": "Spanish", "fa": "Farsi", "fr": "French", "ht": "Haitian Creole", + "hi": "Hindi", "id": "Indonesian", "it": "Italian", "ja": "Japanese", + "ko": "Korean", "nl": "Dutch", "pl": "Polish", "pt": "Portuguese", + "ru": "Russian", "so": "Somali", "sw": "Swahili", "tl": "Tagalog", + "th": "Thai", "tr": "Turkish", "uk": "Ukrainian", "ur": "Urdu", + "vi": "Vietnamese", "zh": "Chinese", +} + + +# ── Languages ───────────────────────────────────────────────────────────────── + +def url_language(url: str) -> str: + """The language code a URL is marked with, or "" when it has none. + + Looks at the first path segment (/es/..., /pt-br/...) and then at a + lang, language or locale query parameter. + """ + locale = url_locale(url) + if locale: + return locale + query = parse_qs(urlparse(url).query) + for key in ("lang", "language", "locale"): + if query.get(key): + return query[key][0].lower()[:5] + return "" + + +def language_groups(urls: list[str]) -> dict[str, int]: + """{language code: count} over the URLs marked with a language.""" + counts: dict[str, int] = {} + for url in urls: + lang = url_language(url) + if lang: + counts[lang] = counts.get(lang, 0) + 1 + return counts + + +def language_label(code: str) -> str: + if not code: + return "No language in the URL (usually the site's main language)" + name = _LANG_NAMES.get(code.split("-")[0]) + return f"{name} ({code})" if name else code + + +def _languages_by_count(urls: list[str]) -> list[tuple[str, int]]: + """Every language present, "" (no marker) included, most pages first.""" + counts: dict[str, int] = {} + for url in urls: + lang = url_language(url) + counts[lang] = counts.get(lang, 0) + 1 + return sorted(counts.items(), key=lambda kv: -kv[1]) + + +# ── Helpers ─────────────────────────────────────────────────────────────────── def _label(url: str, max_len: int = 72) -> str: parsed = urlparse(url) - path = parsed.path or "/" - label = f"{parsed.netloc}{path}" - if len(label) > max_len: - label = label[: max_len - 4] + "…" + label = f"{parsed.netloc}{parsed.path or '/'}" if parsed.query: - label += "[dim]?…[/dim]" + label += f"?{parsed.query}" + if len(label) > max_len: + label = label[: max(8, max_len - 1)] + "…" return label -def _page_count(total: int) -> int: - return max(1, (total + _PAGE_SIZE - 1) // _PAGE_SIZE) +def _page_count(total: int, page_size: int = _PAGE_SIZE) -> int: + return max(1, (total + page_size - 1) // page_size) class _URLReviewer: - """Stateful paginated URL review session.""" + """State of one review: the full list, the shown subset, the ticks and the cursor.""" def __init__(self, urls: list[str]) -> None: self._all: list[str] = list(urls) # full discovered set (never mutated) @@ -78,46 +163,86 @@ def __init__(self, urls: list[str]) -> None: self._selected: set[str] = set(urls) # selected URLs (mutable) self._page: int = 0 # 0-indexed current page self._filter: str = "" # active filter pattern + self._lang: str | None = None # language shown; None = all + self._cursor: int = 0 # index into self._view + self._page_size: int = _PAGE_SIZE self._detail: str = "" # extra block for the next render (help, inspect) + self._languages = [code for code, _ in _languages_by_count(self._all)] - # ── Display ─────────────────────────────────────────────────────────────── + # ── View ────────────────────────────────────────────────────────────────── def _page_slice(self) -> list[str]: - start = self._page * _PAGE_SIZE - return self._view[start : start + _PAGE_SIZE] - - def _render(self, message: str = "", notice: str = "") -> None: - """Draw the whole review screen: table, then messages, then key hints.""" - page_urls = self._page_slice() - total_pages = _page_count(len(self._view)) - n_selected = len(self._selected & set(self._view)) - - filter_hint = f" · filter: {self._filter}" if self._filter else "" - ui.screen( - "Choose pages to scrape", - f"page {self._page + 1}/{total_pages} · {n_selected}/{len(self._view)} ticked{filter_hint}", - ) + start = self._page * self._page_size + return self._view[start : start + self._page_size] + + def _recompute_view(self, pattern: str) -> list[str]: + view = self._all + if self._lang is not None: + view = [u for u in view if url_language(u) == self._lang] + if pattern: + view = filter_urls(view, pattern, base_domain=None) + return view + + def _set_view(self, view: list[str]) -> None: + self._view = view + self._page = 0 + self._cursor = 0 + + def set_page_size(self, size: int) -> None: + """Fit the page to the terminal, keeping the cursor on screen.""" + size = max(3, size) + if size != self._page_size: + self._page_size = size + self._page = self._cursor // size + + def move(self, delta: int) -> None: + if not self._view: + return + self._cursor = max(0, min(len(self._view) - 1, self._cursor + delta)) + self._page = self._cursor // self._page_size + + def turn_page(self, delta: int) -> str: + total = _page_count(len(self._view), self._page_size) + target = self._page + delta + if not 0 <= target < total: + return "Already on the last page." if delta > 0 else "Already on the first page." + self._page = target + self._cursor = min(target * self._page_size, max(0, len(self._view) - 1)) + return "" - table = Table(box=box.SIMPLE, show_header=True, header_style="bold") - table.add_column("#", style="dim", width=4, justify="right") - table.add_column("✓", width=3, justify="center") - table.add_column("URL", no_wrap=True) - - start = self._page * _PAGE_SIZE - for i, url in enumerate(page_urls, start=1): - row_num = start + i - check = "[green]✓[/]" if url in self._selected else "[dim]·[/]" - table.add_row(str(row_num), check, _label(url)) - - ui.console.print(table) - if notice: - ui.warn(notice) - if self._detail: - ui.console.print(self._detail) - self._detail = "" - if message: - ui.info(message) - ui.hints(_HELP_TEXT) + def current(self) -> str | None: + return self._view[self._cursor] if self._view else None + + def toggle_current(self) -> None: + url = self.current() + if url is None: + return + if url in self._selected: + self._selected.discard(url) + else: + self._selected.add(url) + + def toggle_all_shown(self) -> str: + shown = set(self._view) + if shown <= self._selected: + self._selected -= shown + return f"Unticked all {len(shown)} shown URLs." + self._selected |= shown + return f"Ticked all {len(shown)} shown URLs." + + def cycle_language(self) -> str: + if len(self._languages) < 2: + return "All URLs are in one language." + order: list[str | None] = [None, *self._languages] + self._lang = order[(order.index(self._lang) + 1) % len(order)] + self._set_view(self._recompute_view(self._filter)) + if self._lang is None: + return "Showing all languages." + return f"Showing {language_label(self._lang)}: {len(self._view)} URLs. Press a to tick/untick them all." + + def keep_language(self, lang: str) -> None: + """Tick only the URLs in *lang*; the others stay listed, unticked.""" + self._selected = {u for u in self._all if url_language(u) == lang} # ── Command handlers ────────────────────────────────────────────────────── @@ -133,20 +258,22 @@ def _resolve_rows(self, spec: str) -> list[int]: except ValueError: return [] - def _do_filter(self, pattern: str) -> str: + def _do_filter(self, pattern: str, live: bool = False) -> str: + """Apply a filter. A typed command that matches nothing is not applied; + while typing (``live``) the empty result is shown so the user sees why.""" if not pattern: - self._view = list(self._all) self._filter = "" - self._page = 0 - return f"Filter cleared — showing all {len(self._all)} URLs." + self._set_view(self._recompute_view("")) + return f"Filter cleared, showing {len(self._view)} URLs." - filtered = filter_urls(self._all, pattern, base_domain=None) - if not filtered: - return f"Pattern '{pattern}' matched 0 URLs — filter not applied." + filtered = self._recompute_view(pattern) + if not filtered and not live: + return f"Pattern '{pattern}' matched 0 URLs, filter not applied." - self._view = filtered self._filter = pattern - self._page = 0 + self._set_view(filtered) + if not filtered: + return f"No URLs match '{pattern}'." return f"Filter matched {len(filtered)} / {len(self._all)} URLs." def _do_toggle(self, spec: str, select: bool) -> str: @@ -154,12 +281,9 @@ def _do_toggle(self, spec: str, select: bool) -> str: if not indices: return f"Invalid row spec: '{spec}'" changed = 0 - start = self._page * _PAGE_SIZE - page_urls = self._page_slice() for idx in indices: - local = idx - start - if 0 <= local < len(page_urls): - url = page_urls[local] + if 0 <= idx < len(self._view): + url = self._view[idx] if select: self._selected.add(url) else: @@ -168,70 +292,58 @@ def _do_toggle(self, spec: str, select: bool) -> str: action = "Selected" if select else "Deselected" return f"{action} {changed} URL(s)." - def _do_inspect(self, spec: str) -> str: - try: - n = int(spec.strip()) - except ValueError: - return f"Invalid row number: '{spec}'" - start = self._page * _PAGE_SIZE - local = n - 1 - start - page_urls = self._page_slice() - if not (0 <= local < len(page_urls)): - return f"Row {n} not on current page." - url = page_urls[local] + def _inspect(self, index: int) -> None: + url = self._view[index] parsed = urlparse(url) - # Shown on the next redraw, so it survives the screen clear. + # Shown on the next redraw, above the key hints. self._detail = ( - f"[bold]URL #{n}[/]\n" - f" Full URL: [cyan]{url}[/]\n" - f" Scheme: {parsed.scheme}\n" + f"URL #{index + 1}\n" + f" Full URL: {url}\n" f" Host: {parsed.netloc}\n" f" Path: {parsed.path or '/'}\n" f" Query: {parsed.query or '(none)'}\n" - f" Selected: {'[green]Yes[/]' if url in self._selected else '[dim]No[/]'}" + f" Language: {language_label(url_language(url))}\n" + f" Ticked: {'yes' if url in self._selected else 'no'}" ) + + def _do_inspect(self, spec: str) -> str: + try: + n = int(spec.strip()) + except ValueError: + return f"Invalid row number: '{spec}'" + if not 1 <= n <= len(self._view): + return f"No row {n}." + self._inspect(n - 1) return "" def handle(self, raw: str) -> tuple[str, bool]: - """Process one command. Returns (message, done). + """Process one typed command. Returns (message, done). - done=True means the user typed 'done' and we should exit the loop. - done=None means the user typed 'q' — returned as (msg, None). + done=True means the user typed 'done'; done=None means 'back'. """ cmd = raw.strip().lower() if not cmd: return "", False - # Navigation if cmd in ("n", "next"): - total = _page_count(len(self._view)) - if self._page < total - 1: - self._page += 1 - return "", False - return "Already on the last page.", False - + return self.turn_page(1), False if cmd in ("p", "prev"): - if self._page > 0: - self._page -= 1 - return "", False - return "Already on the first page.", False + return self.turn_page(-1), False try: pg = int(cmd) - total = _page_count(len(self._view)) + total = _page_count(len(self._view), self._page_size) if 1 <= pg <= total: - self._page = pg - 1 + self.turn_page(pg - 1 - self._page) return "", False return f"Page must be between 1 and {total}.", False except ValueError: pass - # Filter if cmd == "f" or cmd.startswith("f "): pattern = raw.strip()[1:].strip() return self._do_filter(pattern), False - # Select / deselect if cmd.startswith("x "): return self._do_toggle(cmd[2:], select=False), False if cmd.startswith("+ "): @@ -244,11 +356,9 @@ def handle(self, raw: str) -> tuple[str, bool]: self._selected -= set(self._view) return f"Deselected all {len(self._view)} URLs.", False - # Inspect if cmd.startswith("i "): return self._do_inspect(cmd[2:]), False - # Done / quit if cmd in ("done", "d"): return "", True if cmd in ("q", "quit", "back", "b"): @@ -258,7 +368,7 @@ def handle(self, raw: str) -> tuple[str, bool]: self._detail = _HELP_FULL return "", False - return f"Unknown command '{raw.strip()}' — type ? for help.", False + return f"Unknown command '{raw.strip()}', press ? for help.", False def selected_urls(self) -> list[str]: """Return selected URLs in original discovery order.""" @@ -266,58 +376,313 @@ def selected_urls(self) -> list[str]: return sorted(self._selected, key=lambda u: order.get(u, 0)) -async def run_url_review(urls: list[str], notice: str = "") -> list[str]: - """Interactive URL review: paginated browser with filter, select, and inspect. +# ── Screen ──────────────────────────────────────────────────────────────────── - The screen is redrawn after every command instead of appended to, so the - table, messages and key hints stay in the same place. ``notice`` (e.g. a - mixed-language warning) is shown on every redraw. +class _Screen: + """The full-screen review list. ``run`` returns "done" or "back".""" - Returns the user-approved subset in original discovery order. - """ - if not urls: - return [] + def __init__(self, reviewer: _URLReviewer, notice: str = "", message: str = "") -> None: + self.r = reviewer + self.notice = notice + self.message = message + self.mode: str | None = None # None, "filter" or "command" + self.input = Buffer(multiline=False, on_text_changed=self._on_input_changed) + self.app = self._build() - reviewer = _URLReviewer(urls) + # Input line (filter / command) - session: PromptSession[str] = PromptSession(style=_PT_STYLE) + def _on_input_changed(self, _buf: Buffer) -> None: + if self.mode == "filter": + self.message = self.r._do_filter(self.input.text.strip(), live=True) - msg = "" - while True: - reviewer._render(msg, notice) + def _open_input(self, mode: str) -> None: + self.mode = mode + self.input.text = self.r._filter if mode == "filter" else "" + self.input.cursor_position = len(self.input.text) + self.app.layout.focus(self.input) + def _close_input(self) -> None: + self.mode = None + self.app.layout.focus(self.list_window) + + # Rendering + + def _text(self): + r = self.r try: - raw = await session.prompt_async(" review> ") - except (KeyboardInterrupt, EOFError): - # Ctrl-C / Ctrl-D → treat as back to menu - ui.info("URL review cancelled — returning to menu.") - return [] + size = get_app().output.get_size() + rows, cols = size.rows, size.columns + except Exception: + rows, cols = 40, 100 + notice_lines = self.notice.count("\n") + 1 if self.notice else 0 + detail_lines = r._detail.count("\n") + 1 if r._detail else 0 + r.set_page_size(rows - _CHROME_LINES - notice_lines - detail_lines) + + total_pages = _page_count(len(r._view), r._page_size) + shown = set(r._view) + n_ticked = len(r._selected & shown) + parts: list[tuple[str, str]] = [] + add = parts.append + + sub = f"page {r._page + 1}/{total_pages} · {n_ticked}/{len(r._view)} ticked" + if len(r._view) != len(r._all): + sub += f" · {len(r._selected)}/{len(r._all)} ticked overall" + if r._lang is not None: + sub += f" · language: {language_label(r._lang) if r._lang else 'no marker'}" + if r._filter: + sub += f" · filter: {r._filter}" + add(("class:title", "DataForge Choose pages to scrape")) + add(("class:subtitle", f" {sub}\n")) + add(("class:rule", "─" * max(10, cols - 1) + "\n")) + if self.notice: + add(("class:notice", self.notice + "\n")) + + num_w = len(str(len(r._view))) + 1 + add(("class:header", f" {'#':>{num_w}} ✓ URL\n")) + start = r._page * r._page_size + page_urls = r._page_slice() + if not page_urls: + add(("class:untick", " (no URLs to show: change the filter with / or press l)\n")) + label_w = max(20, cols - num_w - 10) + for i, url in enumerate(page_urls): + idx = start + i + ticked = url in r._selected + line_style = "class:cursor" if idx == r._cursor else "" + pointer = "›" if idx == r._cursor else " " + add((line_style, f"{pointer} ")) + add((f"{line_style} class:rownum", f"{idx + 1:>{num_w}} ")) + add((f"{line_style} {'class:tick' if ticked else 'class:untick'}", "✓" if ticked else "·")) + add((line_style, f" {_label(url, label_w)}")) + add(("", "\n")) + + # Pager: the arrows show which way there are more URLs. + has_prev, has_next = r._page > 0, r._page < total_pages - 1 + add(("class:pager" if has_prev else "class:pager.off", " ◀ ← prev")) + add(("class:subtitle", f" page {r._page + 1} of {total_pages} ")) + add(("class:pager" if has_next else "class:pager.off", "next → ▶\n")) + + if r._detail: + add(("class:detail", r._detail + "\n")) + if self.message: + add(("class:message", self.message + "\n")) + add(("class:rule", "─" * max(10, cols - 1) + "\n")) + add(("class:hint", _HINTS)) + return parts + + def _prompt_text(self): + label = "filter> " if self.mode == "filter" else "command> " + return [("class:prompt", f" {label}")] + + # Keys + + def _build(self) -> Application: + kb = KeyBindings() + browsing = Condition(lambda: self.mode is None) + typing = Condition(lambda: self.mode is not None) + + def act(fn): + """Run a list action: clear the one-shot detail and message first.""" + def handler(event) -> None: + self.r._detail = "" + self.message = "" + fn(event) + return handler + + @kb.add("up", filter=browsing) + @kb.add("k", filter=browsing) + @act + def _(event) -> None: + self.r.move(-1) + + @kb.add("down", filter=browsing) + @kb.add("j", filter=browsing) + @act + def _(event) -> None: + self.r.move(1) + + @kb.add("space", filter=browsing) + @act + def _(event) -> None: + self.r.toggle_current() + self.r.move(1) + + @kb.add("right", filter=browsing) + @kb.add("pagedown", filter=browsing) + @act + def _(event) -> None: + self.message = self.r.turn_page(1) + + @kb.add("left", filter=browsing) + @kb.add("pageup", filter=browsing) + @act + def _(event) -> None: + self.message = self.r.turn_page(-1) + + @kb.add("home", filter=browsing) + @act + def _(event) -> None: + self.r.move(-len(self.r._view)) + + @kb.add("end", filter=browsing) + @act + def _(event) -> None: + self.r.move(len(self.r._view)) + + @kb.add("a", filter=browsing) + @act + def _(event) -> None: + self.message = self.r.toggle_all_shown() + + @kb.add("l", filter=browsing) + @act + def _(event) -> None: + self.message = self.r.cycle_language() + + @kb.add("i", filter=browsing) + @act + def _(event) -> None: + if self.r._view: + self.r._inspect(self.r._cursor) + + @kb.add("?", filter=browsing) + @act + def _(event) -> None: + self.r._detail = _HELP_FULL + + @kb.add("/", filter=browsing) + @act + def _(event) -> None: + self._open_input("filter") + + @kb.add(":", filter=browsing) + @act + def _(event) -> None: + self._open_input("command") + + @kb.add("enter", filter=browsing) + def _(event) -> None: + event.app.exit(result="done") + + @kb.add("escape", filter=browsing) + @kb.add("q", filter=browsing) + @kb.add("c-c") + def _(event) -> None: + event.app.exit(result="back") + + @kb.add("enter", filter=typing) + def _(event) -> None: + text = self.input.text + mode = self.mode + self._close_input() + if mode == "command": + self.r._detail = "" + self.message, done = self.r.handle(text) + if done is True: + event.app.exit(result="done") + elif done is None: + event.app.exit(result="back") + + @kb.add("escape", filter=typing) + def _(event) -> None: + if self.mode == "filter": + self.message = self.r._do_filter("") + self._close_input() + + self.list_window = Window( + FormattedTextControl(self._text, focusable=True, show_cursor=False), + wrap_lines=False, + ) + input_row = ConditionalContainer( + VSplit([ + Window(FormattedTextControl(self._prompt_text), dont_extend_width=True), + Window(BufferControl(buffer=self.input), height=1), + ]), + filter=typing, + ) + return Application( + layout=Layout(HSplit([self.list_window, input_row]), focused_element=self.list_window), + key_bindings=kb, + style=_PT_STYLE, + full_screen=True, + mouse_support=False, + ) + + async def run(self) -> str: + return await self.app.run_async() or "back" + - msg, done = reviewer.handle(raw) +# ── Entry point ─────────────────────────────────────────────────────────────── - if done is None: +async def _ask_language(reviewer: _URLReviewer) -> str | None: + """On a site in several languages, ask which one to tick. + + Returns a notice for the review screen, or None if the user backed out. + """ + groups = _languages_by_count(reviewer._all) + if len(groups) < 2: + return "" + choices = [ + questionary.Choice(f"{language_label(code)}: {n} pages", value=code) + for code, n in groups + ] + choices.append(questionary.Choice(f"All languages: {len(reviewer._all)} pages", value="__all__")) + picked = await questionary.select( + "This site has pages in several languages. Which should be ticked?", + choices=choices, + instruction="(the others stay in the list, unticked)", + style=_QSTYLE, + ).ask_async() + if picked is None: + return None + if picked == "__all__": + return "Several languages found. Press l to look at one language at a time." + reviewer.keep_language(picked) + return (f"Only {language_label(picked)} pages are ticked. " + "Press l to see the other languages, a to tick all shown.") + + +_QSTYLE = QStyle([ + ("qmark", "fg:cyan bold"), ("question", "bold"), + ("answer", "fg:cyan bold"), ("pointer", "fg:cyan bold"), +]) + + +async def run_url_review(urls: list[str], notice: str = "", ask_language: bool = True) -> list[str]: + """Let the user choose which discovered URLs to scrape. + + ``notice`` is shown at the top of the list on every redraw. With + ``ask_language`` off (browsing only), a multilingual site is not asked + about. Returns the ticked URLs in discovery order, or [] if the user + backed out. + """ + if not urls: + return [] + + reviewer = _URLReviewer(urls) + lang_notice = await _ask_language(reviewer) if ask_language else "" + if lang_notice is None: + ui.info("Returning to menu.") + return [] + notice = "\n".join(n for n in (notice, lang_notice) if n) + + message = "" + while True: + result = await _Screen(reviewer, notice, message).run() + if result != "done": ui.info("Returning to menu.") return [] - if done: - selected = reviewer.selected_urls() - total = len(urls) - picked = len(selected) - - if picked == 0: - msg = "No URLs ticked yet. Tick some with + or all, then type done." - continue - - _QSTYLE = QStyle([ - ("qmark", "fg:cyan bold"), ("question", "bold"), - ("answer", "fg:cyan bold"), ("pointer", "fg:cyan bold"), - ]) - confirmed = await questionary.confirm( - f"Scrape {picked} of {total} page{'s' if picked != 1 else ''}?", - default=True, - style=_QSTYLE, - ).ask_async() - - if confirmed: - return selected - msg = "Selection kept. Keep editing, then type done." + selected = reviewer.selected_urls() + if not selected: + message = "No URLs ticked yet. Tick some with space (or a for all shown), then press enter." + continue + + picked, total = len(selected), len(urls) + confirmed = await questionary.confirm( + f"Scrape {picked} of {total} page{'s' if total != 1 else ''}?", + default=True, + style=_QSTYLE, + ).ask_async() + if confirmed: + return selected + message = "Selection kept. Keep editing, then press enter." diff --git a/tests/test_autosave.py b/tests/test_autosave.py index 07ad567..381227e 100644 --- a/tests/test_autosave.py +++ b/tests/test_autosave.py @@ -46,4 +46,4 @@ def test_checkpoint_skipped_when_autosave_disabled(tmp_settings): session = db.get(PipelineSession, ctx.session_id) # _init_session records the output folder (#40); with autosave off the # checkpoint must add none of its counters. - assert set(session.config()) == {"output_dir"} + assert set(session.config()) == {"output_dir", "discovery_scope"} diff --git a/tests/test_quality_and_lang.py b/tests/test_quality_and_lang.py index 9684980..b36f8dc 100644 --- a/tests/test_quality_and_lang.py +++ b/tests/test_quality_and_lang.py @@ -9,7 +9,7 @@ # ── Language group detection ─────────────────────────────────────────────────── # Import the private helper directly from app module -from dataforge.cli.app import _detect_language_groups +from dataforge.cli.url_review import language_groups as _detect_language_groups from dataforge.storage import DataFormat, SyntheticSample, open_session diff --git a/tests/test_scope_and_saved_keys.py b/tests/test_scope_and_saved_keys.py new file mode 100644 index 0000000..ca5b798 --- /dev/null +++ b/tests/test_scope_and_saved_keys.py @@ -0,0 +1,139 @@ +"""Discovery scope (just this page / follow links / whole site) and global API keys.""" +from __future__ import annotations + +import os +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from dataforge.agents import ExplorerAgent, PipelineContext +from dataforge.storage import DataFormat, URLSource + + +def _ctx(scope: str) -> PipelineContext: + settings = MagicMock() + settings.max_crawl_pages = 10 + settings.max_crawl_depth = 2 + return PipelineContext( + session_id="s", session_name="s", goal="", format=DataFormat.qa, + seed_urls=["https://example.com/guide/start"], settings=settings, + discovery_scope=scope, + ) + + +# ── Discovery scope ─────────────────────────────────────────────────────────── + +@pytest.mark.asyncio +async def test_page_scope_uses_the_seed_without_any_request(): + client = MagicMock() + client.get_safe = AsyncMock() + with patch("dataforge.agents.explorer.discover_sitemap_urls", new=AsyncMock()) as sitemaps, \ + patch("dataforge.agents.explorer.crawl", new=AsyncMock()) as crawl: + urls, source = await ExplorerAgent(_ctx("page"))._explore_seed( + client, "https://example.com/guide/start") + assert urls == ["https://example.com/guide/start"] + assert source == URLSource.manual + sitemaps.assert_not_called() + crawl.assert_not_called() + client.get_safe.assert_not_called() + + +@pytest.mark.asyncio +async def test_links_scope_crawls_from_the_seed_and_skips_the_sitemap(): + found = ["https://example.com/guide/start", "https://example.com/guide/next"] + with patch("dataforge.agents.explorer.discover_sitemap_urls", new=AsyncMock()) as sitemaps, \ + patch("dataforge.agents.explorer.crawl", new=AsyncMock(return_value=found)) as crawl: + urls, source = await ExplorerAgent(_ctx("links"))._explore_seed( + MagicMock(), "https://example.com/guide/start") + assert urls == found + assert source == URLSource.crawl + sitemaps.assert_not_called() + assert crawl.call_args.args[1] == "https://example.com/guide/start" + + +@pytest.mark.asyncio +async def test_site_scope_still_reads_the_sitemap(): + with patch("dataforge.agents.explorer.discover_sitemap_urls", + new=AsyncMock(return_value=["https://example.com/sitemap.xml"])), \ + patch("dataforge.agents.explorer.parse_sitemaps", + new=AsyncMock(return_value=["https://example.com/guide/start/a"])): + urls, source = await ExplorerAgent(_ctx("site"))._explore_seed( + MagicMock(), "https://example.com/guide/start") + assert urls == ["https://example.com/guide/start/a"] + assert source == URLSource.sitemap + + +# ── Globally saved API keys ─────────────────────────────────────────────────── + +@pytest.fixture +def clean_env(monkeypatch, tmp_path): + for var in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY", "GEMINI_API_KEY", "GROQ_API_KEY", + "TOGETHER_API_KEY", "DATAFORGE_LLM_PROVIDER", "DATAFORGE_LLM_MODEL"): + monkeypatch.delenv(var, raising=False) + monkeypatch.chdir(tmp_path) + return tmp_path + + +def _saved(keys: dict[str, str], prefs: dict | None = None): + """Patch the global config: *keys* as stored in the OS keychain.""" + from dataforge.cli import prefs as user_prefs + return ( + patch.object(user_prefs, "get_api_key", side_effect=lambda k: keys.get(k, "")), + patch.object(user_prefs, "load", return_value=prefs or {}), + ) + + +def test_keychain_key_is_loaded_even_when_a_project_env_exists(clean_env): + """The bug: a key saved globally lives in the keychain, and .env existing + (which `dataforge config` always creates) skipped loading it.""" + from dataforge.cli.preflight import apply_saved_config + + (clean_env / ".env").write_text("DATAFORGE_LLM_PROVIDER=anthropic\n") + a, b = _saved({"ANTHROPIC_API_KEY": "sk-ant-global"}) + with a, b: + apply_saved_config(clean_env / ".env") + assert os.environ["ANTHROPIC_API_KEY"] == "sk-ant-global" + + +def test_env_file_key_wins_over_global_and_is_exported(clean_env): + from dataforge.cli.preflight import apply_saved_config + + (clean_env / ".env").write_text("OPENAI_API_KEY=sk-project\n") + a, b = _saved({"OPENAI_API_KEY": "sk-global"}) + with a, b: + apply_saved_config(clean_env / ".env") + assert os.environ["OPENAI_API_KEY"] == "sk-project" + + +def test_process_env_wins_over_everything(clean_env, monkeypatch): + from dataforge.cli.preflight import apply_saved_config + + monkeypatch.setenv("OPENAI_API_KEY", "sk-shell") + a, b = _saved({"OPENAI_API_KEY": "sk-global"}) + with a, b: + apply_saved_config(clean_env / ".env") + assert os.environ["OPENAI_API_KEY"] == "sk-shell" + + +def test_saved_provider_does_not_override_project_env(clean_env): + from dataforge.cli.preflight import apply_saved_config + + (clean_env / ".env").write_text("DATAFORGE_LLM_PROVIDER=groq\n") + a, b = _saved({}, {"llm_provider": "anthropic", "llm_model": "claude-x"}) + with a, b: + apply_saved_config(clean_env / ".env") + assert "DATAFORGE_LLM_PROVIDER" not in os.environ + assert os.environ["DATAFORGE_LLM_MODEL"] == "claude-x" + + +def test_credentials_check_uses_the_saved_key_instead_of_prompting(clean_env): + from dataforge.cli import preflight + + settings = MagicMock(llm_provider="anthropic", anthropic_api_key="") + a, b = _saved({"ANTHROPIC_API_KEY": "sk-ant-global"}) + with a, b, patch.object(preflight, "get_settings", return_value=settings), \ + patch.object(preflight.getpass, "getpass") as ask: + ok, missing = preflight.check_llm_credentials() + assert ok and missing is None + ask.assert_not_called() + assert os.environ["ANTHROPIC_API_KEY"] == "sk-ant-global" diff --git a/tests/test_url_review.py b/tests/test_url_review.py index 2122fd2..f2efab9 100644 --- a/tests/test_url_review.py +++ b/tests/test_url_review.py @@ -1,11 +1,15 @@ """Tests for the interactive URL review module and related helpers.""" from __future__ import annotations +import contextlib import uuid from pathlib import Path from unittest.mock import AsyncMock, patch import pytest +from prompt_toolkit.application import create_app_session +from prompt_toolkit.input import create_pipe_input +from prompt_toolkit.output import DummyOutput from dataforge.collectors.sitemap import filter_urls from dataforge.storage.database import init_db, open_session, persist_url_selection @@ -272,7 +276,18 @@ def test_reviewer_unknown_command(): assert "Unknown command" in msg -# ── run_url_review integration (mocked PromptSession + questionary.confirm) ─── +# ── run_url_review integration (keys sent through a pipe input) ───────────── + +DOWN, UP, ENTER = "\x1b[B", "\x1b[A", "\r" + + +@contextlib.contextmanager +def keys(text: str): + """Run the review screen against *text* as typed keys.""" + with create_pipe_input() as inp, create_app_session(input=inp, output=DummyOutput()): + inp.send_text(text) + yield + @pytest.mark.asyncio async def test_run_url_review_empty_input(): @@ -282,103 +297,114 @@ async def test_run_url_review_empty_input(): @pytest.mark.asyncio -async def test_run_url_review_done_and_confirm(): - """done command + confirm=True returns selected URLs.""" +async def test_run_url_review_enter_and_confirm(): + """Enter + confirm returns every URL (all start ticked).""" from dataforge.cli.url_review import run_url_review urls = ["https://example.com/a", "https://example.com/b"] - - with ( - patch("dataforge.cli.url_review.PromptSession") as MockSession, - patch("dataforge.cli.url_review.questionary") as mock_q, - ): - MockSession.return_value.prompt_async = AsyncMock(return_value="done") + with keys(ENTER), patch("dataforge.cli.url_review.questionary") as mock_q: mock_q.confirm.return_value.ask_async = AsyncMock(return_value=True) - result = await run_url_review(urls) - - assert set(result) == set(urls) + assert result == urls @pytest.mark.asyncio -async def test_run_url_review_deselect_then_done(): - """Deselecting a URL before done excludes it from result.""" +async def test_run_url_review_space_unticks_row(): + """Space unticks the row under the cursor and moves down.""" from dataforge.cli.url_review import run_url_review - urls = ["https://example.com/a", "https://example.com/b"] - commands = ["x 1", "done"] - call_idx = 0 - - async def next_cmd(*_a, **_kw): - nonlocal call_idx - cmd = commands[call_idx % len(commands)] - call_idx += 1 - return cmd - - with ( - patch("dataforge.cli.url_review.PromptSession") as MockSession, - patch("dataforge.cli.url_review.questionary") as mock_q, - ): - MockSession.return_value.prompt_async = next_cmd + urls = ["https://example.com/a", "https://example.com/b", "https://example.com/c"] + # untick a (cursor moves to b), skip b, untick c + with keys(" " + DOWN + " " + ENTER), patch("dataforge.cli.url_review.questionary") as mock_q: mock_q.confirm.return_value.ask_async = AsyncMock(return_value=True) - result = await run_url_review(urls) - assert result == ["https://example.com/b"] @pytest.mark.asyncio -async def test_run_url_review_ctrl_c_cancels(): - """Ctrl-C / KeyboardInterrupt during prompt returns empty list.""" +async def test_run_url_review_filter_then_untick_all_shown(): + """/ filters as you type; a unticks everything shown.""" from dataforge.cli.url_review import run_url_review - urls = ["https://example.com/a"] - - with patch("dataforge.cli.url_review.PromptSession") as MockSession: - MockSession.return_value.prompt_async = AsyncMock(side_effect=KeyboardInterrupt) + urls = ["https://example.com/blog/a", "https://example.com/products/b"] + with keys("/blog" + ENTER + "a" + ENTER), patch("dataforge.cli.url_review.questionary") as mock_q: + mock_q.confirm.return_value.ask_async = AsyncMock(return_value=True) result = await run_url_review(urls) - - assert result == [] + assert result == ["https://example.com/products/b"] @pytest.mark.asyncio -async def test_run_url_review_filter_then_done(): - """Filter command narrows the view; done returns only filtered+selected set.""" +async def test_run_url_review_typed_command(): + """: opens the command line for the typed commands (x 1-2).""" from dataforge.cli.url_review import run_url_review - urls = ["https://example.com/blog/a", "https://example.com/products/b"] - commands = ["f /blog/*", "done"] - call_idx = 0 - - async def next_cmd(*_a, **_kw): - nonlocal call_idx - cmd = commands[call_idx % len(commands)] - call_idx += 1 - return cmd - - with ( - patch("dataforge.cli.url_review.PromptSession") as MockSession, - patch("dataforge.cli.url_review.questionary") as mock_q, - ): - MockSession.return_value.prompt_async = next_cmd + urls = ["https://example.com/a", "https://example.com/b", "https://example.com/c"] + with keys(":x 1-2" + ENTER + ENTER), patch("dataforge.cli.url_review.questionary") as mock_q: mock_q.confirm.return_value.ask_async = AsyncMock(return_value=True) - result = await run_url_review(urls) + assert result == ["https://example.com/c"] - # The filter changes the view but selected still includes all — only the - # confirm step matters. Selection is never auto-removed by filtering. - assert "https://example.com/blog/a" in result + +@pytest.mark.asyncio +async def test_run_url_review_q_returns_empty(): + from dataforge.cli.url_review import run_url_review + with keys("q"): + result = await run_url_review(["https://example.com/a"]) + assert result == [] @pytest.mark.asyncio -async def test_run_url_review_quit_returns_empty(): - """q command returns empty list (user aborted).""" +async def test_run_url_review_ctrl_c_cancels(): from dataforge.cli.url_review import run_url_review + with keys("\x03"): + result = await run_url_review(["https://example.com/a"]) + assert result == [] - urls = ["https://example.com/a"] - with patch("dataforge.cli.url_review.PromptSession") as MockSession: - MockSession.return_value.prompt_async = AsyncMock(return_value="q") - result = await run_url_review(urls) +@pytest.mark.asyncio +async def test_run_url_review_language_choice_ticks_one_language(): + """On a multilingual site, the language picked up front is the only one ticked.""" + from dataforge.cli.url_review import run_url_review - assert result == [] + urls = [ + "https://example.com/floods", + "https://example.com/fires", + "https://example.com/es/inundaciones", + "https://example.com/fr/inondations", + ] + with keys(ENTER), patch("dataforge.cli.url_review.questionary") as mock_q: + mock_q.select.return_value.ask_async = AsyncMock(return_value="") + mock_q.confirm.return_value.ask_async = AsyncMock(return_value=True) + result = await run_url_review(urls) + assert result == ["https://example.com/floods", "https://example.com/fires"] + + +# ── keyboard helpers on _URLReviewer ────────────────────────────────────────── + +def test_reviewer_turn_page_moves_cursor(): + r = make_reviewer([f"https://example.com/p{i}" for i in range(65)]) + r.turn_page(1) + assert r._page == 1 and r._cursor == 30 + assert r.turn_page(5) == "Already on the last page." + r.turn_page(-1) + assert r._cursor == 0 + + +def test_reviewer_cycle_language(): + r = make_reviewer([ + "https://example.com/a", "https://example.com/b", "https://example.com/es/a", + ]) + r.cycle_language() # most common first: no marker + assert r._view == ["https://example.com/a", "https://example.com/b"] + r.cycle_language() # then es + assert r._view == ["https://example.com/es/a"] + r.cycle_language() # back to all + assert len(r._view) == 3 + + +def test_url_language(): + from dataforge.cli.url_review import url_language + assert url_language("https://example.com/es/terremotos") == "es" + assert url_language("https://example.com/pt-br/x") == "pt-br" + assert url_language("https://example.com/page?lang=fr") == "fr" + assert url_language("https://example.com/us/news") == "" From 263b04aa588e35eb255b7d52dfd3efe6a4e4db61 Mon Sep 17 00:00:00 2001 From: "Ian T." Date: Thu, 24 Sep 2026 16:44:04 -0700 Subject: [PATCH 3/3] chore: bump version to 2.7.0 pyproject.toml, uv.lock, CITATION.cff (version 2.7.0), README current-release line, and a CHANGELOG 2.7.0 section. Minor release: keyboard URL review with a language choice, a scope choice to scrape only the given page, its links, or the whole site, and a fix for a globally saved API key not being found. --- CHANGELOG.md | 6 ++++++ CITATION.cff | 2 +- README.md | 15 +++++++-------- pyproject.toml | 2 +- uv.lock | 2 +- 5 files changed, 16 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aa8624..8a6497e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,12 @@ Format loosely follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +## [2.7.0] - 2026-09-24 + +Easier URL selection: pick pages from the keyboard, choose the language on a +multilingual site, and scrape just the page you gave instead of the whole +site. Also fixes an API key saved globally not being found. + ### Added - **Scrape just the page you gave**: after entering URLs that are not a sitemap, the wizard asks whether to scrape only those pages (no discovery, diff --git a/CITATION.cff b/CITATION.cff index 3f67f97..9d9ca47 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -12,7 +12,7 @@ authors: orcid: 'https://orcid.org/0009-0000-4888-1941' affiliation: William Jessup University repository-code: 'https://github.com/ianktoo/data-forge' -version: 2.6.0 +version: 2.7.0 date-released: '2026-09-24' abstract: >- DataForge is an open-source pipeline for building synthetic diff --git a/README.md b/README.md index 1c8e500..54a8271 100644 --- a/README.md +++ b/README.md @@ -6,14 +6,13 @@ [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://www.python.org/) [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) -**Current release: 2.6.0.** An easier interactive menu: tasks in plain -words, one screen per step, help at every stage, and "export what I have so -far" at any point (discovered URLs, pages, chunks or samples). A missing or -invalid API key now stops the run with one clear message instead of retrying -every chunk. `dataforge scrape` saves pages and their tables with no AI, API -key or cost. Tested on Linux, Windows and macOS with Python 3.11 to 3.14, -with standalone binaries for Windows, Linux and Apple Silicon Macs. See the -[changelog](CHANGELOG.md#260---2026-09-24). +**Current release: 2.7.0.** Easier URL selection: tick pages from the +keyboard (arrows, Space, Left and Right to turn pages, `/` to filter), choose +which language to keep on a multilingual site, and scrape just the page you +gave, its links, or the whole site. A key saved with "Save API key globally" +is now found in every folder. Tested on Linux, Windows and macOS with Python +3.11 to 3.14, with standalone binaries for Windows, Linux and Apple Silicon +Macs. See the [changelog](CHANGELOG.md#270---2026-09-24). Fine-tuning needs data, and good domain data is trapped in documentation sites, knowledge bases and public archives. Getting it out usually means diff --git a/pyproject.toml b/pyproject.toml index bb6e25b..879ae1d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "llm-web-crawler" -version = "2.6.0" +version = "2.7.0" description = "LLM data collection and synthetic fine-tuning dataset pipeline" readme = "README.md" requires-python = ">=3.11" diff --git a/uv.lock b/uv.lock index 147529c..e3c6445 100644 --- a/uv.lock +++ b/uv.lock @@ -1559,7 +1559,7 @@ wheels = [ [[package]] name = "llm-web-crawler" -version = "2.6.0" +version = "2.7.0" source = { editable = "." } dependencies = [ { name = "aiofiles" },