Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ Format loosely follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

## [2.7.1] - 2026-09-24

Live feedback while scraping, a Back option that goes back, and a way to
build a dataset from pages you already scraped.

### Added
- **Use a scrape folder as dataset input**: *Build an AI training dataset*
now offers **Pages I scraped earlier** next to the URL options. Pick a
Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ authors:
orcid: 'https://orcid.org/0009-0000-4888-1941'
affiliation: William Jessup University
repository-code: 'https://github.com/ianktoo/data-forge'
version: 2.7.0
version: 2.7.1
date-released: '2026-09-24'
abstract: >-
DataForge is an open-source pipeline for building synthetic
Expand Down
15 changes: 8 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,14 @@
[![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Current release: 2.7.0.** Easier URL selection: tick pages from the
keyboard (arrows, Space, Left and Right to turn pages, `/` to filter), choose
which language to keep on a multilingual site, and scrape just the page you
gave, its links, or the whole site. A key saved with "Save API key globally"
is now found in every folder. Tested on Linux, Windows and macOS with Python
3.11 to 3.14, with standalone binaries for Windows, Linux and Apple Silicon
Macs. See the [changelog](CHANGELOG.md#270---2026-09-24).
**Current release: 2.7.1.** Scraping several URLs now shows each page as
it is fetched, Back works in every menu, and the Multiple URLs box finishes on
Enter at an empty line. New: build a dataset from pages you already scraped
with no AI, without fetching them again. 2.7.0 added keyboard URL review, a
language choice on multilingual sites, and scraping just the page you gave.
Tested on Linux, Windows and macOS with Python 3.11 to 3.14, with standalone
binaries for Windows, Linux and Apple Silicon Macs. See the
[changelog](CHANGELOG.md#271---2026-09-24).

Fine-tuning needs data, and good domain data is trapped in documentation
sites, knowledge bases and public archives. Getting it out usually means
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "llm-web-crawler"
version = "2.7.0"
version = "2.7.1"
description = "LLM data collection and synthetic fine-tuning dataset pipeline"
readme = "README.md"
requires-python = ">=3.11"
Expand Down
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading