[sadik9]

Summary

wikimapper no longer works with the current Wikimedia dumps:

1. wikimapper download fails with HTTP Error 403: Forbidden, because dumps.wikimedia.org rejects the default Python-urllib/3.x user agent.
2. wikimapper create silently creates an empty index. Since 2025 the SQL dumps put every row on its own line, and the old parser only read the INSERT INTO line. With the current English dump it finishes with exit code 0 and 0 rows.

This PR fixes both. The patched processor.py parses both the old dump format (all rows on the INSERT INTO ... VALUES line) and the new format (one row per line), with the same code path. It produces the same index as before: same table, same indexes, same redirect handling.

Every change in this PR was reproduced first with real dumps or a real run (output below). Things that turned out not to be a problem were left unchanged; see Investigated, but not changed.

Results with the current English Wikipedia dump

Dumps: enwiki-latest-{page,page_props,redirect}.sql.gz from October 2026 (2.4 GB, 484 MB and 188 MB).

	master	this PR
wikimapper download enwiki-latest	HTTPError: HTTP Error 403: Forbidden	downloads all three files
wikimapper create enwiki-latest	exit code 0, 0 rows	19,315,587 rows, 19,280,223 with a Wikidata ID
Time for create (ASUS ROG Zephyrus G14, GA403UV)	94 s (parses nothing)	481 s
wikimapper title2id index_enwiki-latest.db Germany	nothing	Q183

The index was built twice, with two stages of this branch, and both builds were identical row by row (0 differing rows in either direction).

The dump format change

master looks for the rows on the INSERT line itself:

def _get_values(line):
    return line[line.find("` VALUES ") + 9 :]

Old format (what master expects): all rows of an INSERT statement on one line.

INSERT INTO `page` VALUES (1,0,'Main_Page',1,0,0.325947813174,...),(2,2,'Purodha',0,0,...),...;

New format (current dumps): INSERT INTO ... VALUES alone on a line, then one row per line.

INSERT INTO `page` VALUES
(1,0,'Main_Page',1,0,0.325947813174,'20240125183010','20240125183006',763159,38,'wikitext',NULL),
(2,2,'Purodha',0,0,0.299155934835,'20260118125525','20260118125523',463342,39924,'wikitext',NULL),
...

On the new format, find(" VALUES ")finds nothing on theINSERTline (it ends withVALUES\n), and the row lines don't start with INSERT INTO`, so they are skipped. Nothing is parsed and no error is raised.

Changes

1. wikimapper/processor.py: parse both dump formats

The rows are matched with regular expressions on their leading columns. That works the same whether a line holds one row or thousands. Only the rows that are needed match at all: namespace 0 for pages and redirects, wikibase_item for page properties. Everything else is skipped by the regex engine.

# A quoted MySQL string with backslash escapes, e.g. 'D\'_boarische_Woocha'. Written as runs of
# unescaped characters (`[^'\\]*`) separated by escapes, which is much faster than a per-character
# alternation like `(?:[^'\\]|\\.)*`
_STRING = r"'([^'\\]*(?:\\.[^'\\]*)*)'"

# Rows are matched by their leading columns only, which works for one and for many rows per line.
# Quotes inside of strings are always escaped, so a match cannot start inside of a string.
# Only the rows that are needed match at all, so all other rows are skipped by the regex engine.
# https://www.mediawiki.org/wiki/Manual:Page_table: page_id, page_namespace = 0, page_title, ...
# Filter the namespace; only use real articles https://www.mediawiki.org/wiki/Manual:Namespace
_PAGE_ROW = re.compile(r"\((\d+),0," + _STRING)
# https://www.mediawiki.org/wiki/Manual:Page_props_table: pp_page, pp_propname, pp_value, ...
# The page property table contains many properties, we only care about the Wikidata id
_PAGE_PROPS_ROW = re.compile(r"\((\d+),'wikibase_item'," + _STRING)
# https://www.mediawiki.org/wiki/Manual:Redirect_table: rd_from, rd_namespace = 0, rd_title, ...
# We only care about redirects to main articles
_REDIRECT_ROW = re.compile(r"\((\d+),0," + _STRING)


def _data_lines(path: str, errors: str) -> Iterator[str]:
    """Yields the lines of `path` that contain rows of an INSERT statement.

    Older dumps put all rows of an INSERT statement on the same line as `INSERT INTO ... VALUES`,
    newer ones (2025 onwards) put every row on its own line. Both layouts are supported.
    """
    with gzip.open(path, "rt", encoding="utf-8", errors=errors, newline="\n") as f:
        for line in f:
            if line.startswith("(") or line.startswith("INSERT INTO"):
                yield line


def _parse_pages(path: str) -> Iterator[Tuple[int, str]]:
    for line in _data_lines(path, errors="strict"):
        for page_id, title in _PAGE_ROW.findall(line):
            yield int(page_id), _unescape(title)

create_index keeps its signature and the database layout. The parsed rows go straight into executemany, and redirects are resolved in SQL, with the same rule as before (a redirect gets its target's Wikidata ID if the target has one):

conn.executemany(
    """UPDATE mapping SET wikidata_id = COALESCE(
        (SELECT target.wikidata_id FROM mapping AS target WHERE target.wikipedia_title = ?),
        wikidata_id
    ) WHERE wikipedia_id = ?""",
    _parse_redirects(redirects_dump),
)

The file decoding is unchanged: page is decoded strictly, and page_props and redirect with errors="ignore" (#12). In the current English dump, page and redirect contain no invalid UTF-8, while page_props has 74,333 lines with binary (gzip-compressed templatedata) values.

The csv/ctypes code is gone, and with it the csv.field_size_limit workaround for #7.

Both formats produce the same index. I rewrote the current barwiki dumps into the old one-line format and built the index from both:

barwiki	rows	with a Wikidata ID	differences
master, old format	44,337	44,273	(reference)
this PR, old format	44,337	44,273	0
this PR, new format	44,337	44,273	0

For simplewiki (400,693 pages) in the old format, the parsed rows are identical to those of master's parser.

Two failure modes that were silent before now raise an error. Both were reproduced with real barwiki dumps:

• A missing dump file: master deleted the existing index first and only failed at the missing redirect dump, after the pages and page properties were already in. A good index (44,273 mapped) was replaced by a broken one: 27,312 mapped, redirects not resolved (Brezel → None), no idx_wikidata_id index. The three dumps are now checked before anything is deleted:

  for dump in [pages_dump, page_props_dump, redirects_dump]:
      if not os.path.isfile(dump):
          raise FileNotFoundError("Dump [{0}] does not exist".format(dump))

• No pages parsed: this raises ValueError("No pages could be parsed from [...], the dump format may have changed") instead of silently writing an empty index.

Parsing speed

Parsing time added on top of reading and decompressing the dump, full English dump (66 M page rows, 46 M page_props rows, 15.5 M redirect rows). All variants produce exactly the same rows:

	regex, filter in Python	regex, filter in the pattern (this PR)	csv module	string scanning (str.find)
page	96 s	29 s	78 s	30 s
page_props	71 s	15 s	27 s	25 s
redirect	26 s	11 s	12 s	13 s
total	193 s	55 s	117 s	68 s

The csv and string-scanning variants only handle one row per line. For the old format (all rows on one line), page in simplewiki: this PR 0.37 s, master's csv parser 2.98 s, string scanning 4.23 s. Most of the remaining build time is resolving the redirects in SQLite (about 300 of the 481 s).

2. wikimapper/download.py: User-Agent, mirror URLs, timeout

User-Agent (HTTP 403). The Wikimedia User-Agent policy blocks generic user agents:

$ curl -I -A "Python-urllib/3.12" https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page.sql.gz
HTTP/1.1 403 Forbidden
$ curl -I -A "wikimapper/0.2.1 (https://github.com/jcklie/wikimapper)" https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page.sql.gz
HTTP/1.1 200 OK

urlretrieve cannot set headers for a single request. Installing a global opener (urllib.request.install_opener) would work, but I checked it: it replaced the opener of the program using wikimapper, so that program's own later requests lost their headers. So the request is sent with its own header, and the rest is what urlretrieve does: 8 KiB blocks, the same progress hook, a temporary file that is only moved to the target when complete, and ContentTooShortError for incomplete downloads.

USER_AGENT = "wikimapper/{0} (https://github.com/jcklie/wikimapper)".format(__version__)

request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(request, timeout=_TIMEOUT) as response:
    with tempfile.NamedTemporaryFile(delete=False) as f:
        ...
if read < total_size:
    raise urllib.error.ContentTooShortError(...)
shutil.move(local_name, target)

Mirror URLs with a path. urljoin drops the last path segment of a mirror URL without a trailing slash. With a real mirror:

--mirror https://mirror.accum.se/mirror/wikimedia.org/dumps
master:    https://mirror.accum.se/mirror/wikimedia.org/barwiki/20261001/...       -> HTTP 404
this PR:   https://mirror.accum.se/mirror/wikimedia.org/dumps/barwiki/20261001/... -> all three files downloaded

# Not `urljoin`, as that drops the last path segment of mirrors like `https://host/dumps`
url = mirror.rstrip("/") + "/" + wiki_name + "/" + date + "/"

Timeout. During testing, a download from dumps.wikimedia.org hung: the connection was established and the temporary file was created, but it stayed at 0 bytes for more than 10 minutes, because urlopen had no timeout. urlopen now gets timeout=60. That applies to each network operation, not to the whole download, so large dumps are not affected.

3. Tests

Test	What it checks
test_create_index_parses_dump_layouts[one row per line]	new format: escaped quotes, namespaces, redirects, a (…) inside a property value
test_create_index_parses_dump_layouts[many rows per line]	the same rows in the old format give the same index
test_create_index_fails_for_empty_dump	no rows parsed → ValueError
test_create_index_fails_for_missing_dump	missing dump → FileNotFoundError, existing index untouched
test_download_request (4 cases)	user agent header; mirror URLs with and without a path and trailing slash
test_download_times_out_on_stalled_server	a local server that stops sending → socket.timeout, no file left behind

These tests need no network access. Run against master, the new-format, empty-dump and missing-dump tests fail and the download tests can't even import (USER_AGENT doesn't exist there). The old-format test passes on master too, as it should. In addition, tests/test_mapper.py defined test_wikipedia_id_to_title twice, so the 15 cases of the first definition never ran. The second one is renamed to test_title_to_wikipedia_id.

master:   59 tests (57 in test_mapper.py)
this PR:  83 passed, including the network tests against the current barwiki and ndswiki dumps

4. Packaging, CI, docs

• setup.py: python_requires=">=3.6" instead of ">=3.5.0", and the 3.5 classifier is removed. The code uses f-strings (cli.py); vermin reports a minimum of 3.6, for both master and this PR.
• .github/workflows/run_tests.yml: Python 3.7 is removed from the matrix. ubuntu-latest is Ubuntu 24.04 (runner images), and the actions/python-versions manifest has no 3.7 build for it, so that job cannot start. 3.8 and 3.9 are available on both Ubuntu 24.04 and Windows.
• README.rst:
  • virtualenv … --no-site-packages: the option was removed in virtualenv 20 (virtualenv: error: unrecognized arguments: --no-site-packages).
  • tox is removed: there is no tox.ini/setup.cfg/pyproject.toml, so tox runs 0 tests and still reports congratulations :).
  • The CLI help shows options:, as printed by Python 3.10+, instead of optional arguments:.
  • The FAQ describes the new parser.
  • "Create your own index" adds the build time for the current English dump.
• Makefile: the html target is removed; it runs cd docs && make html, and there is no docs folder.

Investigated, but not changed

These looked suspicious, but checking them with real data showed no problem, so they are left as they are:

• Redirects to other wikis (rd_interwiki set, e.g. 'de') are looked up as local titles, as before. The current English dump has none of them (all 15,551,857 rows have an empty rd_interwiki), and in barwiki (52) and simplewiki (141) none changes the index.
• Truncated downloads: master already raises ContentTooShortError and never moves an incomplete file to the target (tested with a server that closes the connection early). This PR keeps that behavior.
• setup.py warnings (test_suite, tests_require, license classifier) don't affect the build, and find_packages(exclude="tests") already produces a wheel without the tests.
• The README usage examples still match the current English index (Germany → Q183, wikipedia_id_to_title(3342) → Bundesrepublik_Deutschland, …). The one exception is id_to_titles("Q183"): Wikipedia has changed since then, so Land_der_Dichter_und_Denker now redirects to Culture_of_Germany and Jerman is its own article.

Notes for the maintainer

• publish_to_pypi.yml uses Python 3.8 on ubuntu-latest and pypa/gh-action-pypi-publish@master. I didn't change it, because I can't test the release pipeline.
• The version string 0.2.1-dev is normalized to 0.2.1.dev0 by current setuptools (just a warning).
• The precomputed indices linked in the README are from April 2019.

🤖 Generated with Claude Code