Release 2.6.0

CHANGELOG.md

@@ -5,6 +5,55 @@ Format loosely follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [2.6.0] - 2026-09-24
+
+A simpler interactive menu with a fixed screen layout, help at every stage and
+export at any stage, plus a fix for runs that looked like an endless error
+loop when the API key was missing or invalid. One behaviour change: Ctrl+C at
+a stage checkpoint now stops and saves the run.
+
+### Added
+- **Export at any stage** (`exporters/stage_data.py`). Discovered URLs (txt,
+  csv, json), scraped pages (one Markdown file per page with front matter,
+  plain text, json, jsonl), chunks (jsonl, json, txt) and all samples (jsonl,
+  json). "Export what I have so far" at a checkpoint offers whatever exists,
+  plus the training set when there are samples. `dataforge export <id>` uses
+  the same flow and no longer refuses a session without samples; Browse my
+  data has an export entry too.
+- **Help**: "What happens next?" at each stage checkpoint explains every
+  stage and the next one. A Help page in the main menu; `help` and `h` work in
+  the URL review.
+- "Scrape pages (no AI)" in the main menu runs `dataforge scrape`.
+
+### Changed
+- **Main menu worded as tasks**: Scrape pages (no AI), Build an AI training
+  dataset, Continue a paused project, Browse my data, Export data, Settings
+  and tools, Help, Clear the screen, Exit. Rarely used items (provider and
+  key, system info, update, uninstall, project list) moved to a submenu.
+  Every choice shows a one-line description.
+- **Screen layout**: wizard steps, pipeline stages and the URL review start on
+  a fresh screen, in one order (title, content, messages, then key hints
+  directly above the prompt). The URL review redraws in place and the URL
+  list is no longer printed twice. `DATAFORGE_NO_CLEAR=1` keeps all output.
+- The interactive menu shows WARNING and above on the terminal unless
+  `DATAFORGE_LOG_LEVEL` is set; the log file still records INFO.
+- **Input**: paths and URLs accept surrounding quotes (Windows "Copy as
+  path"), PowerShell drag-and-drop (`& 'C:\...'`), curly quotes, `<url>`,
+  and URL lists separated by commas, spaces or lines.
+- **Behaviour change**: Ctrl+C at a stage checkpoint stops and saves the run;
+  it previously continued to the next stage.
+
+### Fixed
+- **A missing or invalid API key no longer loops.** Fatal credential and
+  connection errors now reach the generator and streaming agent instead of
+  being swallowed per chunk; auth failures are not retried; LiteLLM debug
+  banners are suppressed; `generate_batch` stops dispatching and cancels
+  pending work on the first fatal error. With 40 chunks and a bad key, the run
+  stops in about 3 seconds with one message, where it previously made 3 calls
+  per chunk.
+- "Export what I have so far" after collection or processing wrote nothing,
+  because it exported only samples.
+
 ## [2.5.0] - 2026-09-23
 
 Quick scrape (#69, first part): `dataforge scrape` and the MCP tool

CITATION.cff

@@ -12,8 +12,8 @@ authors:
     orcid: 'https://orcid.org/0009-0000-4888-1941'
     affiliation: William Jessup University
 repository-code: 'https://github.com/ianktoo/data-forge'
-version: 2.5.0
-date-released: '2026-09-23'
+version: 2.6.0
+date-released: '2026-09-24'
 abstract: >-
   DataForge is an open-source pipeline for building synthetic
   instruction-tuning datasets from websites. It honors HTTP

README.md

@@ -6,13 +6,14 @@
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://www.python.org/)
 [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
-**Current release: 2.5.0.** New: `dataforge scrape` saves pages and their
-tables (CSV, JSON) with no AI, API key or cost, also available to AI agents as
-the MCP tool `scrape_page`. Discovery is faster and politer: a crawl without a
-sitemap follows site navigation, visits the most useful pages first and
-downloads each page once. Tested on Linux, Windows and macOS with Python 3.11
-to 3.14, with standalone binaries for Windows, Linux and Apple Silicon Macs.
-See the [changelog](CHANGELOG.md#250---2026-09-23).
+**Current release: 2.6.0.** An easier interactive menu: tasks in plain
+words, one screen per step, help at every stage, and "export what I have so
+far" at any point (discovered URLs, pages, chunks or samples). A missing or
+invalid API key now stops the run with one clear message instead of retrying
+every chunk. `dataforge scrape` saves pages and their tables with no AI, API
+key or cost. Tested on Linux, Windows and macOS with Python 3.11 to 3.14,
+with standalone binaries for Windows, Linux and Apple Silicon Macs. See the
+[changelog](CHANGELOG.md#260---2026-09-24).
 
 Fine-tuning needs data, and good domain data is trapped in documentation
 sites, knowledge bases and public archives. Getting it out usually means

docs/CONFIGURATION.md

@@ -23,7 +23,8 @@ Run `dataforge config` to set your provider and API key interactively.
 | `DATAFORGE_MAX_CRAWL_DEPTH` | `3` | Max link depth for the fallback crawl (the seed is depth 0) |
 | `DATAFORGE_CHUNK_SIZE` | `512` | Tokens per chunk |
 | `DATAFORGE_CHUNK_OVERLAP` | `64` | Token overlap between chunks |
-| `DATAFORGE_LOG_LEVEL` | `INFO` | `DEBUG` / `INFO` / `WARNING` / `ERROR` |
+| `DATAFORGE_LOG_LEVEL` | `INFO` | `DEBUG` / `INFO` / `WARNING` / `ERROR`. Terminal level; the interactive menu shows `WARNING` and above unless this is set. The log file always records `INFO`. |
+| `DATAFORGE_NO_CLEAR` | unset | Set to `1` to keep all output instead of clearing the screen between steps |
 | `DATAFORGE_OUTPUT_DIR` | `./output` | Session output directory (logs also stored here in `logs/`) |
 | `DATAFORGE_DB_PATH` | `./dataforge.db` | SQLite database path |
 | `DATAFORGE_AUTOSAVE` | `true` | Checkpoint progress to the session DB after every stage |

docs/TECHNICAL.tex

@@ -357,6 +357,11 @@ \subsection{Export}
 version~2.4.2 the Unsloth file had no lineage at all (project issue
 \#19), so a consumer training from it lost the ability to cite a sample's
 source document.
+Earlier stages are exportable too, without an LLM: the discovered URL
+list (text, CSV, JSON), the scraped pages (one Markdown or plain-text file
+per page, or JSON), and the chunks with their source URL (JSON, JSONL,
+text), so a session that stops before generation still yields a usable
+corpus.
 
 \subsubsection{Dataset introspection}
 \label{sec:introspection}

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "llm-web-crawler"
-version = "2.5.0"
+version = "2.6.0"
 description = "LLM data collection and synthetic fine-tuning dataset pipeline"
 readme = "README.md"
 requires-python = ">=3.11"