Repository navigation
Release 2.7.0 - #75
Merged
Merged
Conversation
The .dataforge file lists local sessions with absolute database and output paths from the user's machine, so it should not be committed.
…ix global API key URL review (cli/url_review.py) is now a full-screen keyboard list: arrow keys move, Space ticks or unticks, Left and Right turn pages (the pager shows arrows where there are more URLs), a ticks or unticks everything shown, / filters as you type, i shows details, Enter finishes, Esc goes back. The page fits the terminal. The typed commands still work after ':'; row numbers now refer to the whole list. On a site with pages in several languages (/es/..., ?lang=fr), the review first asks which language to tick; the others stay listed, unticked, and l cycles the list through one language at a time. Language detection moves from app.py to url_review.py and reuses the recipe's locale list. Discovery scope: for seed URLs that are not a sitemap, the wizard asks whether to scrape only those pages (discovery and review skipped), follow their links (crawl from the seeds without the sitemap), or take the whole site. PipelineContext.discovery_scope carries the choice; it is saved in the session config so resume repeats it. Page scope ignores skip_known. Fix: a key saved with "Save API key globally" lives in the OS keychain, but startup read only prefs.json and only when the folder had no .env, which `dataforge config` always creates, so runs still asked for the key. preflight.apply_saved_config now loads saved keys, provider and model with the precedence shell env, then .env, then global config; the credentials check looks in the keychain before prompting; and a provider changed from the menu updates the loaded settings. Tests: key-press tests for the review screen, scope tests for the explorer, and saved-key precedence tests. TECHNICAL.tex discovery section updated.
pyproject.toml, uv.lock, CITATION.cff (version 2.7.0), README current-release line, and a CHANGELOG 2.7.0 section. Minor release: keyboard URL review with a language choice, a scope choice to scrape only the given page, its links, or the whole site, and a fix for a globally saved API key not being found.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 2.7.0: a minor release, because it adds a keyboard URL review, a language choice and a discovery scope choice to the interactive wizard, and fixes a globally saved API key not being found. No breaking changes to commands, recipes or file formats. One behaviour change in the review's typed commands: row numbers (
x 3-8,i 3) now refer to the whole list, not only the current page.Version bump
pyproject.tomlanduv.lock: 2.6.0 to 2.7.0CITATION.cff:version: 2.7.0[2.7.0] - 2026-09-24section with a summary, Added, Changed and FixedWhat 2.7.0 contains (37fc86b, branch
feat/url-review-scope)Keyboard URL review (
cli/url_review.py)aticks or unticks everything shown,/filters as you type,ishows details,?help, Enter finishes, Esc/q goes back.:.Language choice
/es/or/pt-br/, or alang/language/localequery parameter), the review first asks which language to tick. The others stay listed, unticked.lcycles the list through one language at a time.app.pytourl_review.pyand reuses the recipe's locale list, so it no longer mistakes any two-letter segment (for example/us/) for a language.Discovery scope
PipelineContext.discovery_scopecarries the choice; it is saved in the session config, so resume repeats it instead of crawling the site. Page scope ignores "skip already scraped".site).Fix: globally saved API key not found
prefs.json, and only when the folder had no.env, whichdataforge configalways creates. So the run still asked for the key.preflight.apply_saved_confignow loads saved keys, provider and model whether or not.envexists. Precedence: shell environment, then.env, then global config. Keys found in.envare exported to the environment too.Tests and docs
tests/test_scope_and_saved_keys.py(scope behaviour of the explorer, key precedence, no prompt when a saved key exists). The review tests now drive the screen with real key presses through a pipe input.docs/TECHNICAL.texdiscovery section describes the scope choice and the language step.scripts/smoke_test.py --mcp --e2e(24/24).This PR also carries a2eb374 (ignore the local
.dataforgesession index), which was on local master.After merge
Tag
v2.7.0on master. The release workflows run the full suite, publish to PyPI, build and smoke-test the standalone binaries, and create the GitHub release;release-smoke.ymlthen tests the PyPI release on every OS and Python.