Skip to content

Release 2.7.0 - #75

Merged
ianktoo merged 4 commits into
masterfrom
release/v2.7.0
Sep 24, 2026
Merged

ianktoo merged 4 commits into
masterfrom
release/v2.7.0

Conversation

@ianktoo

@ianktoo ianktoo commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Release 2.7.0: a minor release, because it adds a keyboard URL review, a language choice and a discovery scope choice to the interactive wizard, and fixes a globally saved API key not being found. No breaking changes to commands, recipes or file formats. One behaviour change in the review's typed commands: row numbers (x 3-8, i 3) now refer to the whole list, not only the current page.

Version bump

  • pyproject.toml and uv.lock: 2.6.0 to 2.7.0
  • CITATION.cff: version: 2.7.0
  • README: current-release line; changelog link points at 2.7.0
  • CHANGELOG: new [2.7.0] - 2026-09-24 section with a summary, Added, Changed and Fixed

What 2.7.0 contains (37fc86b, branch feat/url-review-scope)

Keyboard URL review (cli/url_review.py)

  • Full-screen list: arrow keys (or j/k) move, Space ticks or unticks and moves down, Left/Right or PgUp/PgDn turn pages, Home/End jump, a ticks or unticks everything shown, / filters as you type, i shows details, ? help, Enter finishes, Esc/q goes back.
  • The pager shows arrows (dimmed when there is nothing that way); the page size follows the terminal height.
  • The typed commands of earlier versions still work after :.

Language choice

  • On a site with pages in several languages (first path segment such as /es/ or /pt-br/, or a lang/language/locale query parameter), the review first asks which language to tick. The others stay listed, unticked. l cycles the list through one language at a time.
  • Detection moves from app.py to url_review.py and reuses the recipe's locale list, so it no longer mistakes any two-letter segment (for example /us/) for a language.

Discovery scope

  • For seed URLs that are not a sitemap, the wizard asks: just these pages (discovery and the review are skipped), deep link search (crawl from the seeds without reading the sitemap), or the whole site (previous behaviour).
  • PipelineContext.discovery_scope carries the choice; it is saved in the session config, so resume repeats it instead of crawling the site. Page scope ignores "skip already scraped".
  • Recipes, headless runs and MCP keep the default (site).

Fix: globally saved API key not found

  • "Save API key globally" stores the key in the OS keychain, but startup read only prefs.json, and only when the folder had no .env, which dataforge config always creates. So the run still asked for the key.
  • preflight.apply_saved_config now loads saved keys, provider and model whether or not .env exists. Precedence: shell environment, then .env, then global config. Keys found in .env are exported to the environment too.
  • The credentials check looks in the keychain before prompting, and a provider changed from the Settings menu updates the loaded settings without a restart.

Tests and docs

  • New tests/test_scope_and_saved_keys.py (scope behaviour of the explorer, key precedence, no prompt when a saved key exists). The review tests now drive the screen with real key presses through a pipe input.
  • docs/TECHNICAL.tex discovery section describes the scope choice and the language step.
  • Locally on Windows: full suite passes; a clean install of the built 2.7.0 wheel passes scripts/smoke_test.py --mcp --e2e (24/24).

This PR also carries a2eb374 (ignore the local .dataforge session index), which was on local master.

After merge

Tag v2.7.0 on master. The release workflows run the full suite, publish to PyPI, build and smoke-test the standalone binaries, and create the GitHub release; release-smoke.yml then tests the PyPI release on every OS and Python.

The .dataforge file lists local sessions with absolute database and output paths from the user's machine, so it should not be committed.
…ix global API key

URL review (cli/url_review.py) is now a full-screen keyboard list: arrow keys
move, Space ticks or unticks, Left and Right turn pages (the pager shows
arrows where there are more URLs), a ticks or unticks everything shown, /
filters as you type, i shows details, Enter finishes, Esc goes back. The page
fits the terminal. The typed commands still work after ':'; row numbers now
refer to the whole list.

On a site with pages in several languages (/es/..., ?lang=fr), the review
first asks which language to tick; the others stay listed, unticked, and l
cycles the list through one language at a time. Language detection moves from
app.py to url_review.py and reuses the recipe's locale list.

Discovery scope: for seed URLs that are not a sitemap, the wizard asks
whether to scrape only those pages (discovery and review skipped), follow
their links (crawl from the seeds without the sitemap), or take the whole
site. PipelineContext.discovery_scope carries the choice; it is saved in the
session config so resume repeats it. Page scope ignores skip_known.

Fix: a key saved with "Save API key globally" lives in the OS keychain, but
startup read only prefs.json and only when the folder had no .env, which
`dataforge config` always creates, so runs still asked for the key.
preflight.apply_saved_config now loads saved keys, provider and model with
the precedence shell env, then .env, then global config; the credentials
check looks in the keychain before prompting; and a provider changed from the
menu updates the loaded settings.

Tests: key-press tests for the review screen, scope tests for the explorer,
and saved-key precedence tests. TECHNICAL.tex discovery section updated.
pyproject.toml, uv.lock, CITATION.cff (version 2.7.0), README current-release line, and a CHANGELOG 2.7.0 section. Minor release: keyboard URL review with a language choice, a scope choice to scrape only the given page, its links, or the whole site, and a fix for a globally saved API key not being found.
@ianktoo
ianktoo merged commit e4cf2ee into master Sep 24, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant