Release 2.7.0

.gitignore

@@ -89,6 +89,7 @@ Thumbs.db
 
 # Project-specific
 dataforge.db
+.dataforge
 run.bat
 run.ps1
 run.sh

CHANGELOG.md

@@ -5,6 +5,40 @@ Format loosely follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [2.7.0] - 2026-09-24
+
+Easier URL selection: pick pages from the keyboard, choose the language on a
+multilingual site, and scrape just the page you gave instead of the whole
+site. Also fixes an API key saved globally not being found.
+
+### Added
+- **Scrape just the page you gave**: after entering URLs that are not a
+  sitemap, the wizard asks whether to scrape only those pages (no discovery,
+  no URL review), follow the links from them (a crawl from the pages, without
+  the sitemap), or take the whole site as before. Resume repeats the same
+  choice.
+- **Language choice in the URL review**: on a site with pages in several
+  languages (`/es/...`, `?lang=fr`), the review first asks which language to
+  tick; the others stay listed, unticked. `l` cycles the list through one
+  language at a time.
+
+### Changed
+- **Keyboard URL review**: arrow keys move, Space ticks or unticks, Left and
+  Right turn pages (the pager shows arrows where there are more URLs), `a`
+  ticks or unticks everything shown, `/` filters as you type, `i` shows
+  details, Enter finishes and Esc goes back. The page fits the terminal. The
+  typed commands (`x 3-8`, `f blog`, ...) still work after `:`; row numbers
+  now refer to the whole list, not only the current page.
+
+### Fixed
+- **Globally saved API key not found**: "Save API key globally" stores the
+  key in the OS keychain, but startup only read `prefs.json`, and only when
+  the folder had no `.env` (which `dataforge config` always creates), so the
+  run still asked for the key. Saved keys, provider and model are now loaded
+  whether or not `.env` exists (shell environment first, then `.env`, then the
+  global config), the key check looks in the keychain before prompting, and a
+  provider changed from the menu takes effect without a restart.
+
 ## [2.6.0] - 2026-09-24
 
 A simpler interactive menu with a fixed screen layout, help at every stage and

CITATION.cff

@@ -12,7 +12,7 @@ authors:
     orcid: 'https://orcid.org/0009-0000-4888-1941'
     affiliation: William Jessup University
 repository-code: 'https://github.com/ianktoo/data-forge'
-version: 2.6.0
+version: 2.7.0
 date-released: '2026-09-24'
 abstract: >-
   DataForge is an open-source pipeline for building synthetic

README.md

@@ -6,14 +6,13 @@
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://www.python.org/)
 [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
-**Current release: 2.6.0.** An easier interactive menu: tasks in plain
-words, one screen per step, help at every stage, and "export what I have so
-far" at any point (discovered URLs, pages, chunks or samples). A missing or
-invalid API key now stops the run with one clear message instead of retrying
-every chunk. `dataforge scrape` saves pages and their tables with no AI, API
-key or cost. Tested on Linux, Windows and macOS with Python 3.11 to 3.14,
-with standalone binaries for Windows, Linux and Apple Silicon Macs. See the
-[changelog](CHANGELOG.md#260---2026-09-24).
+**Current release: 2.7.0.** Easier URL selection: tick pages from the
+keyboard (arrows, Space, Left and Right to turn pages, `/` to filter), choose
+which language to keep on a multilingual site, and scrape just the page you
+gave, its links, or the whole site. A key saved with "Save API key globally"
+is now found in every folder. Tested on Linux, Windows and macOS with Python
+3.11 to 3.14, with standalone binaries for Windows, Linux and Apple Silicon
+Macs. See the [changelog](CHANGELOG.md#270---2026-09-24).
 
 Fine-tuning needs data, and good domain data is trapped in documentation
 sites, knowledge bases and public archives. Getting it out usually means

docs/TECHNICAL.tex

@@ -201,10 +201,15 @@ \subsection{Discovery}
 (\texttt{robots.txt}, rate limit, DataForge's User-Agent), does not load
 images, media or fonts, and is not used at all on a site that declares a
 \texttt{Crawl-delay}, because the sub-requests of a page load cannot be
-spaced one per delay. In interactive use, discovery is followed by a
-review step --- substring, glob, or regex filtering plus per-URL
-selection --- that a recipe replaces declaratively with
-\texttt{source.include} / \texttt{source.exclude} / \texttt{source.max\_urls}.
+spaced one per delay. In interactive use, seed URLs that are not a
+sitemap come with a choice of scope: only the given pages (discovery is
+skipped), the pages they link to (a crawl from the seeds without the
+sitemap), or the whole site. Discovery is then followed by a review
+step: on a site served in several languages the user first picks the one
+to keep, then filters by substring, glob or regular expression and ticks
+URLs from the keyboard. A recipe replaces this step declaratively with
+\texttt{source.language} / \texttt{source.include} /
+\texttt{source.exclude} / \texttt{source.max\_urls}.
 
 \subsection{Collection}
 \label{sec:collection}

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "llm-web-crawler"
-version = "2.6.0"
+version = "2.7.0"
 description = "LLM data collection and synthetic fine-tuning dataset pipeline"
 readme = "README.md"
 requires-python = ">=3.11"

src/dataforge/agents/base.py

@@ -40,6 +40,10 @@ class PipelineContext:
     n_per_chunk: int = 3
     ignore_robots: bool = False
     skip_known: bool = False  # skip URLs already scraped in prior sessions
+    # How far discovery reaches from the seed URLs: "site" reads the site's
+    # sitemap (crawling when there is none), "links" crawls from the seeds
+    # without the sitemap, "page" uses the seeds as given.
+    discovery_scope: str = "site"
     # The recipe's language/include/exclude check. The fallback crawl spends
     # its page budget only on URLs that pass it (#63). None = keep all.
     url_filter: Callable[[str], bool] | None = None

src/dataforge/agents/explorer.py

@@ -49,8 +49,9 @@ async def run(self) -> PipelineContext:
         # Deduplicate (already maintained by source_map insertion order)
         discovered = all_urls
 
-        # Optionally skip URLs already scraped in any previous session
-        if self.ctx.skip_known and discovered:
+        # Optionally skip URLs already scraped in any previous session. Pages
+        # named one by one are what the user asked for, so they are kept.
+        if self.ctx.skip_known and discovered and self.ctx.discovery_scope != "page":
             discovered = self._filter_already_scraped(discovered)
 
         self.ctx.discovered_urls = discovered
@@ -70,8 +71,12 @@ async def _explore_seed(self, client, seed: str) -> tuple[list[str], str]:
             urls = await parse_sitemap(client, seed)
             return (urls if urls else [seed], URLSource.sitemap)
 
+        scope = self.ctx.discovery_scope
+        if scope == "page":
+            return ([seed], URLSource.manual)
+
         # 2. Try to discover sitemaps (every one robots.txt lists, #60)
-        sitemap_urls = await discover_sitemap_urls(client, base)
+        sitemap_urls = [] if scope == "links" else await discover_sitemap_urls(client, base)
         sitemap_url = ", ".join(sitemap_urls)
         if sitemap_urls:
             raw_urls = await parse_sitemaps(client, sitemap_urls)
@@ -104,7 +109,8 @@ async def _explore_seed(self, client, seed: str) -> tuple[list[str], str]:
 
         # 3. Fall back to BFS crawl from seed URL
         self.log.info(
-            f"No usable sitemap for {base}, starting BFS crawl "
+            f"{'Following links from ' + seed if scope == 'links' else 'No usable sitemap for ' + base}, "
+            "starting BFS crawl "
             f"(max_pages={self.ctx.settings.max_crawl_pages}, max_depth={self.ctx.settings.max_crawl_depth})"
         )
         crawled = await crawl(

src/dataforge/agents/orchestrator.py

@@ -212,7 +212,9 @@ def _init_session(self) -> None:
                     seed_urls=json.dumps(self.ctx.seed_urls),
                     # A recipe may set its own output_dir; later commands
                     # (stats, resume) need it to find this session's files.
-                    config_json=json.dumps({"output_dir": str(s.output_dir.resolve())}),
+                    # discovery_scope lets resume repeat discovery the same way.
+                    config_json=json.dumps({"output_dir": str(s.output_dir.resolve()),
+                                            "discovery_scope": self.ctx.discovery_scope}),
                 ))
                 db.commit()
 

src/dataforge/cli/app.py

@@ -426,6 +426,7 @@ def _label(x: PipelineSession) -> str:
         format=DataFormat(session.format),
         seed_urls=session.seed_url_list(),
         settings=s,
+        discovery_scope=session.config().get("discovery_scope", "site"),
     )
 
     # Re-hydrate prior-stage data from DB so _checkpoint() doesn't zero out saved counts
@@ -804,10 +805,16 @@ async def _configure() -> None:
     from dataforge.cli import prefs as user_prefs
     user_prefs.set("llm_provider", provider)
     user_prefs.set("llm_model", model)
+    # Settings is loaded once per process: without this, a change made from
+    # the menu only took effect after a restart, and the next run still
+    # checked the old provider's key.
+    s = get_settings()
+    s.llm_provider = provider
+    s.llm_model = model
 
     if info.requires_key:
         import getpass
-        existing = os.getenv(info.key_env, "")
+        existing = os.getenv(info.key_env, "") or user_prefs.get_api_key(info.key_env)
         masked = f"{existing[:8]}..." if len(existing) > 8 else ("set" if existing else "")
         prompt_label = (
             f"  {info.key_env} [{masked}] (leave blank to keep): "
@@ -825,7 +832,9 @@ async def _configure() -> None:
                 updated = _set_env_var(updated, info.key_env, key_value)
                 env_path.write_text("\n".join(updated) + "\n")
                 ui.success(f"Saved {info.key_env} to {env_path.resolve()}")
-        elif not existing:
+        elif existing:
+            os.environ.setdefault(info.key_env, existing)
+        else:
             ui.info(f"No key entered — set {info.key_env} via 'dataforge config' when ready")
 
     ui.success(f"Saved: provider={provider}, model={model}")
@@ -1463,7 +1472,7 @@ async def _browse_discovered_urls(session_id: str) -> None:
         ui.info("No URLs discovered yet for this session.")
         return
     ui.info(f"[dim]Browsing {len(urls)} discovered URLs — changes are not saved[/]")
-    await run_url_review(urls)  # read-only: return value discarded
+    await run_url_review(urls, ask_language=False)  # read-only: return value discarded
 
 
 # ── Explore menu (interactive data browser) ────────────────────────────────────
@@ -1582,6 +1591,15 @@ async def _step_urls(state: dict) -> StepResult:
         ui.error("No valid URLs provided")
         return "back"
     state["seed_urls"] = result
+    # A sitemap already lists the pages to choose from; anything else could
+    # mean one page or a whole site, so ask rather than always crawling.
+    if all(u.lower().endswith(".xml") for u in result):
+        state["discovery_scope"] = "site"
+    else:
+        scope = await prompts.ask_discovery_scope(len(result))
+        if scope is None:
+            return "back"
+        state["discovery_scope"] = scope
     return "next"
 
 
@@ -1640,8 +1658,11 @@ async def _step_config(state: dict) -> StepResult:
 
         from urllib.parse import urlparse as _urlparse
         seed_domain = _urlparse(state.get("seed_urls", [""])[0]).netloc or "this site"
-        skip_known = await prompts.ask_skip_known(seed_domain)
-        state["skip_known"] = skip_known
+        # Pages named one by one are scraped even if seen before.
+        if state.get("discovery_scope", "site") != "page":
+            state["skip_known"] = await prompts.ask_skip_known(seed_domain)
+        else:
+            state["skip_known"] = False
 
         threshold = await prompts.ask_quality_threshold()
         if threshold is None:
@@ -1776,6 +1797,7 @@ async def _interactive_pipeline() -> None:
             n_per_chunk=state["n_per_chunk"],
             ignore_robots=state.get("ignore_robots", False),
             skip_known=state.get("skip_known", False),
+            discovery_scope=state.get("discovery_scope", "site"),
             quality_threshold=state.get("quality_threshold", 0.5),
             generation_model=state.get("generation_model", ""),
             quality_model=state.get("quality_model", ""),
@@ -1887,23 +1909,6 @@ async def _run_wizard(state: dict) -> str:
     PipelineStage.quality:    PipelineStage.export,
 }
 
-_LANG_RE = re.compile(
-    r"/([a-z]{2}(?:-[a-z]{2})?)/|[?&]lang(?:uage)?=([a-z]{2})|[?&]locale=([a-z]{2})",
-    re.IGNORECASE,
-)
-
-
-def _detect_language_groups(urls: list[str]) -> dict[str, int]:
-    """Return {locale_code: count} for URLs containing language/locale patterns."""
-    counts: dict[str, int] = {}
-    for url in urls:
-        m = _LANG_RE.search(url)
-        if m:
-            lang = next(g for g in m.groups() if g).lower()
-            counts[lang] = counts.get(lang, 0) + 1
-    return counts
-
-
 async def _adjust_settings(context: PipelineContext) -> None:
     """Mid-session settings menu — change the model or output dir between stages.
 
@@ -2006,19 +2011,12 @@ async def post_discovery_hook(stage: str, context: PipelineContext) -> bool:
             ui.warn("Discovery returned 0 URLs. Check that the site has a reachable sitemap or provide a direct sitemap URL.")
             return False
 
-        # The review screen lists the URLs itself; printing them here as well
-        # only pushed the review's key hints off screen.
-        notice = ""
-        lang_groups = _detect_language_groups(context.discovered_urls)
-        if len(lang_groups) >= 2:
-            top = ", ".join(f"{k} ({v})" for k, v in sorted(lang_groups.items(), key=lambda x: -x[1])[:5])
-            notice = (
-                f"Several languages found: {top}. "
-                "To keep one, filter it, e.g. [bold]f /en/[/], then [bold]none[/] / [bold]all[/] as needed."
-            )
-
-        from .url_review import run_url_review
-        selected = await run_url_review(context.discovered_urls, notice=notice)
+        if context.discovery_scope == "page":
+            # The user named the pages; there is nothing to choose from.
+            selected = list(context.discovered_urls)
+        else:
+            from .url_review import run_url_review
+            selected = await run_url_review(context.discovered_urls)
 
         if not selected:
             ui.warn("No URLs selected — returning to menu.")

src/dataforge/cli/preflight.py

@@ -34,28 +34,48 @@
 }
 
 
+def apply_saved_config(env_path: Path | None = None) -> None:
+    """Load the global config into the process environment.
+
+    Precedence: the process environment, then the project's .env, then the
+    global config saved by ``dataforge config``. Keys saved globally live in
+    the OS keychain (prefs.json only when no keychain is available), so both
+    are read. A key found in .env is exported too: litellm reads keys from the
+    environment, not from Settings.
+    """
+    from dotenv import dotenv_values
+
+    from dataforge.cli import prefs as user_prefs
+
+    env_path = env_path or Path(".env")
+    in_file = {k: v for k, v in dotenv_values(env_path).items() if v} if env_path.exists() else {}
+
+    for var, pref in (("DATAFORGE_LLM_PROVIDER", "llm_provider"), ("DATAFORGE_LLM_MODEL", "llm_model")):
+        saved = user_prefs.get(pref)
+        if saved and not os.getenv(var) and var not in in_file:
+            os.environ[var] = saved
+
+    key_envs = {k for k in _PROVIDER_KEY_MAP.values() if k}
+    key_envs.update(user_prefs.load().get("api_keys", {}))
+    for key_env in sorted(key_envs):
+        if os.getenv(key_env):
+            continue
+        value = in_file.get(key_env) or user_prefs.get_api_key(key_env)
+        if value:
+            os.environ[key_env] = value
+
+
 def check_env_file() -> bool:
     """Warn (not fatal) if .env doesn't exist or no provider key is configured."""
     from dataforge.cli import prefs as user_prefs
 
     env_path = Path(".env").resolve()
-    env_exists = env_path.exists()
-    if not env_exists:
+    if not env_path.exists():
         show_warning(
             f".env file not found at {env_path}",
             f"Run:  dataforge config  to save settings globally to {user_prefs._prefs_path()}",
         )
-        # Apply saved prefs as env var defaults so pip-installed users keep their config
-        saved_provider = user_prefs.get("llm_provider")
-        saved_model = user_prefs.get("llm_model")
-        if saved_provider and not os.getenv("DATAFORGE_LLM_PROVIDER"):
-            os.environ["DATAFORGE_LLM_PROVIDER"] = saved_provider
-        if saved_model and not os.getenv("DATAFORGE_LLM_MODEL"):
-            os.environ["DATAFORGE_LLM_MODEL"] = saved_model
-        # Apply globally-saved API keys so pip-installed users work without .env
-        for key_env, value in user_prefs.load().get("api_keys", {}).items():
-            if value and not os.getenv(key_env):
-                os.environ[key_env] = value
+    apply_saved_config(env_path)
 
     # Check whether the active provider is satisfied (keyless like Ollama, or has a key set)
     s = get_settings()
@@ -85,6 +105,11 @@ def check_llm_credentials() -> tuple[bool, str | None]:
         return _check_openai_compatible(s.local_base_url, s.local_api_key)
 
     if key_env and not os.getenv(key_env) and not getattr(s, key_env.lower(), ""):
+        from dataforge.cli import prefs as user_prefs
+        saved = user_prefs.get_api_key(key_env)
+        if saved:
+            os.environ[key_env] = saved
+            return True, None
         # Offer a live prompt before failing
         try:
             value = getpass.getpass(f"  {key_env} not set — paste your key now (hidden, session-only): ")