Repository navigation
Release 2.6.0 - #74
Merged
Merged
Conversation
A missing or invalid key made generation look like an endless error loop: - generate_from_chunk caught MissingCredentialError and LLMConnectionError like any other failure and returned [], so the fatal error never reached the generator or streaming agent. Every remaining chunk tried again. - The tenacity retry retried auth failures 3 times with 2-20s backoff, though they can never succeed. - LiteLLM printed its "Give Feedback / Get Help" banner on every failure. - generate_batch left queued tasks running after one raised. Now fatal errors propagate, auth failures are not retried, LiteLLM debug banners are suppressed, and generate_batch stops dispatching and cancels pending work on the first fatal error. With 40 chunks and a bad key the run stops after the calls already in flight (about 3s) with one message, where it previously made 3 calls per chunk. This also makes the streaming claim in TECHNICAL.tex true: streaming already had a fatal-error branch, but it never fired.
…port at any stage Main menu - Worded as tasks: Scrape pages (no AI), Build an AI training dataset, Continue a paused project, Browse my data, Export data, Settings and tools, Help, Clear the screen, Exit. Rarely used items (provider and key, system info, update, uninstall, project list) moved to a submenu. - Every choice shows a one-line description under the list. - "Scrape pages (no AI)" brings `dataforge scrape` into the menu. Screen layout - Wizard steps, pipeline stages and the URL review start on a fresh screen instead of appending. DATAFORGE_NO_CLEAR=1 keeps all output. - Every screen follows one order: title, content, messages, then key hints directly above the prompt. The URL review redraws in place, and the URL list is no longer printed twice after discovery, which pushed the hints off screen. The mixed-language warning moved into the review screen. - The interactive menu shows WARNING and above on the terminal unless DATAFORGE_LOG_LEVEL is set. The log file still records INFO. Help - "What happens next?" at each stage checkpoint explains every stage and the next one in detail. Help page in the main menu. `help` and `h` work in the URL review. Input - Paths and URLs accept surrounding quotes (Windows "Copy as path"), PowerShell drag-and-drop (& 'C:\...'), curly quotes, <url>, and URL lists separated by commas, spaces or lines. Export at any stage - New exporters/stage_data.py writes discovered URLs (txt, csv, json), scraped pages (one Markdown file per page with front matter, plain text, json, jsonl), chunks (jsonl, json, txt) and all samples (jsonl, json). - "Export what I have so far" at a checkpoint previously exported only samples, so it wrote nothing after collection or processing. It now offers whatever exists, plus the training set when there are samples, then returns to the checkpoint menu. - `dataforge export <id>` uses the same flow and no longer refuses a session without samples. Browse my data has an export entry too. Behaviour change: Ctrl+C at a stage checkpoint now stops and saves the run; it previously continued to the next stage.
pyproject.toml, uv.lock, CITATION.cff (version 2.6.0, date 2026-09-24), README current-release line, and a CHANGELOG 2.6.0 section. Minor release: a simpler interactive menu with export at any stage, and a fix for the endless error loop on a missing or invalid API key. One behaviour change: Ctrl+C at a stage checkpoint now stops and saves the run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 2.6.0: a minor release, because it adds export at any stage and help to the interactive menu, and fixes the API key error loop. No breaking changes to commands or file formats. One behaviour change in the interactive menu: Ctrl+C at a stage checkpoint now stops and saves the run (it previously continued to the next stage).
Version bump
pyproject.tomlanduv.lock: 2.5.0 to 2.6.0CITATION.cff:version: 2.6.0,date-released: 2026-09-24[2.6.0] - 2026-09-24section with a summary, Added, Changed and FixedWhat 2.6.0 contains
Interactive menu (d5a521e, branch
fix/cli-ux)dataforge scrapeinto the menu.DATAFORGE_NO_CLEAR=1keeps all output. Terminal logging in the menu is WARNING and above unlessDATAFORGE_LOG_LEVELis set.help/hin the URL review.<url>, and URL lists split by commas, spaces or lines.exporters/stage_data.py): URLs, pages, chunks and samples in several formats.dataforge export <id>no longer refuses a session without samples.Generation fix (44eba65)
generate_batchcancels pending work on the first fatal error. A bad key with 40 chunks stops in about 3 seconds with one message.docs/TECHNICAL.textrue.Tests
tests/test_input_cleaning.py,tests/test_stage_data_export.py,tests/test_generation_fatal_errors.py. The full suite passes locally on Windows.After merge
Tag
v2.6.0on master. The release workflows run the full suite first, publish to PyPI, smoke-test and upload the standalone binaries, and create the GitHub release;release-smoke.ymlthen tests the PyPI release on every OS and Python.