Skip to content

fix(searxng): search in the install locale and keep Seznam's .cz shops out - #6734

Merged
Lightheartdevs merged 1 commit into
mainfrom
claude/searxng-locale
Sep 26, 2026
Merged

Lightheartdevs merged 1 commit into
mainfrom
claude/searxng-locale

Conversation

@Lightheartdevs

Copy link
Copy Markdown
Collaborator

What was wrong

The fleet's "SearXNG returns Czech results" problem is not a locale setting. On tower1, tower2 and tower3 every major engine refuses the household IP: Google says "access denied", Brave says "too many requests", Startpage shows a CAPTCHA, and DuckDuckGo times out. That leaves Seznam, which #4215 enabled as a fallback, answering every query. Seznam is a Czech-market index, and it ignores language. For GPU queries, most of the page is .cz shops (zbozi.cz, alza.cz, heureka.cz).

Hermes and Perplexica got those results. So would Pixel in PIXEL_WEB_SEARCH_PROVIDER=searxng mode. Pixel on the fleet uses parallel-free, so it is not affected.

A second, smaller problem: SearXNG's default language is auto. Pixel (the OpenClaw searxng plugin), Hermes and Perplexica send no language and no browser Accept-Language header, so auto becomes "all".

What this changes

  • Search language from the install locale. The installers now write search.default_lang:
    • Linux phase 06 uses LC_ALL, then LC_MESSAGES, then LANG.
    • macOS uses the same variables, then the macOS locale.
    • Windows uses Get-Culture.
    • A request that sends its own language still wins.
    • Only tags that the pinned image accepts are written; anything else becomes en. I confirmed on tower2 that an unknown value such as xx-YY stops SearXNG from starting ("Invalid settings.yml").
  • Seznam stays enabled, but its .cz results are dropped unless the locale is Czech. This uses SearXNG's built-in hostnames plugin (hostnames.remove: '\.cz$'). A Czech install gets no rule.
  • Factory template. config/searxng/settings.yml gets default_lang: "en" and the .cz rule.
  • Shared code and docs. The mapping lives once for bash, in installers/lib/searxng-locale.sh, which Linux phase 06 and the macOS generator both use. Windows has a mirror function. The SearXNG README documents the setting.
  • Test. tests/test-searxng-config-contract.sh is extended (see Tests). It was not run by CI before; it now runs in test-linux.yml and make test.

Why not disable Seznam

I replayed the same queries with Seznam turned off: 0 results for all 44 queries. The engines that would replace it are still blocked, so turning Seznam off takes away the English results it does return, such as Songkick, Eventbrite and Ticketmaster for the event queries.

Why not set Pixel's webSearch.language too

The OpenClaw searxng plugin adds language only when the model passes one or plugins.entries.searxng.config.webSearch.language is set. That config is written by the vendored, digest-checked Pixel render-config.mjs. The server-side default already covers Pixel's searxng-mode calls, and Hermes, Perplexica and Open WebUI as well. A second copy of the setting could drift from the first.

Evidence (replay on tower2, throwaway containers, pinned image searxng/searxng:2026.3.8-a563127a2)

Queries. 44 in total. 40 are real fleet search queries from rounds 070-092 and the Hermes comparator runs: a seeded sample of 10 from each of Pixel component-research, Pixel event-search, Hermes component-research and Hermes event-search. The other 4 are general control queries.

How they were sent. Each query went to every configuration back to back, the way the OpenClaw plugin sends it (q and format=json, Accept-Language: *).

Configurations compared:

  • main: today's settings.
  • Seznam off: Seznam disabled, default_lang set.
  • this PR: default_lang: en-US plus the .cz rule. The replay used the equivalent pattern (.*\.)?cz$. The settings actually generated by phase 06 were then booted and re-checked on 4 queries: the .cz results were removed for en_US and kept for cs_CZ.

"Fact hosts" means results on hosts where main's reader already reads a needed fact (from the scout corpus).

main Seznam off this PR
results 402 0 280
Czech results (.cz host, /cs-cz/ path or Czech title) 154 (38%) 0 15 (5%)
other results 248 0 265
queries with any result 40/44 0/44 39/44
results on fact hosts (queries) 16 (9) 0 17 (10)
latency p50 / p95 3003 / 3008 ms 3003 / 3004 ms 3003 / 3023 ms

By journey:

  • Component research: the Czech share drops from 60% (123 of 205) to 8% (7 of 89). Queries with any result drop from 20 to 17; the 3 that go empty were all-Czech before.
  • Event search: Czech results drop from 15 to 5, and the English results are kept. The count went from 157 to 164, which is Seznam's run-to-run variance.

Latency:

  • The paired median difference between main and this PR is 0 ms.
  • The 3 s floor comes from DuckDuckGo timing out on every query, at SearXNG's 3 s request timeout. Every SearXNG search on these hosts pays that today. It is a separate issue and not changed here.

Page reading: this PR does not change the reader, so it makes 0 previously unread needed facts readable. It changes which leads search returns.

Tests

  • tests/test-searxng-config-contract.sh, now in CI:
    • locale mapping cases (en_US.UTF-8 → en-US, de_LU → de, zh-Hant-TW → zh-TW, C/POSIX/empty/* → en, sr_RS@latin → en, and so on) and LC_ALL > LC_MESSAGES > LANG precedence;
    • renders the Linux phase 06 heredoc and the macOS generator for en_US and cs_CZ and requires identical output;
    • renders the Windows generator where PowerShell exists (CI's Ubuntu image has pwsh) and requires the same output;
    • the tag list must name the image pinned in compose.yaml, and the bash and Windows lists must match;
    • a YAML check with PyYAML.
  • Where it passed: Git Bash with Windows PowerShell 5.1 and PyYAML, and on tower2 (Linux, where the PowerShell step is skipped).
  • Mutation checks: 5 injected regressions were each caught: phase 06 without the filter, a Windows tag-list drift, a pin bump, a dropped en-US tag, and a Windows Czech rule typo.
  • Installer tests that run phase 06, on tower2, branch and main: 11 of 11 pass on both. They are installer-env-smoke, phase06-env-rerun-quoting, linux-ollama-port-override, llama-memory-budget, external-services, validate-env, rootless-docker-ownership, external-llm-switchboard-env, installer-contracts, installer-hardening, and the contract test.
  • Generated settings boot: settings for en_US, cs_CZ, de_DE and zh_TW all start healthy in the pinned image.
  • Linters:
    • shellcheck finds nothing at warning level in the new lib and test, and phase 06 and the macOS generator have the same 12 existing warnings as main;
    • PSScriptAnalyzer with the repo settings finds nothing in env-generator.ps1.
  • Pixel plugin JS suite on this head: 3121 tests, 3083 pass, 0 fail.

Risks

  • Search is still thin on blocked hosts. With every major engine blocked, a US GPU query that Seznam answers only with .cz shops now returns few or no results instead of Czech ones. That is more honest, but it is not better search. The real fix is engine availability: Yandex answered with US-relevant results from tower2, or a paid API. That is an owner decision.
  • Czech-language pages on global hosts stay. They are 15 of the 280 results. 9 have a Czech path, such as nvidia.com/cs-cz/ (4) and prohoster.info (2); the rest have Czech titles, such as play.google.com. A hostname rule cannot see paths or titles.
  • Non-Czech owners lose all .cz sites, from every engine. This applies even when Google answers. An owner who wants them can delete the rule from settings.yml.
  • When existing installs get this. Linux and Windows installs regenerate settings.yml on the next installer run. macOS keeps an existing file unless the installer is forced.
  • Keeping the tag list current. The list is the pinned image's sxng_locales. The contract test fails on a pin bump until someone rechecks it. If the list goes stale, SearXNG still starts, because only listed tags are written.
  • Coordination. git merge-tree is clean against feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698 (1f0158f) and fix(pixel): read PDF text in the shared page reader; binary bodies are never page reads #6724 (d7268e0). No files are shared with either.

🤖 Generated with Claude Code

…s out

On tower1, tower2 and tower3 every major SearXNG engine refuses the
household IP (Google "access denied", Brave "too many requests",
Startpage "CAPTCHA", DuckDuckGo times out), so Seznam, enabled in #4215
as the fallback, answers every query. Seznam is a Czech-market index and
ignores the language parameter: for GPU queries most of the page is
.cz shops (zbozi.cz, alza.cz, heureka.cz). That is what Hermes and
Perplexica got, and what searxng-mode Pixel would get.

- Installers write search.default_lang from the install locale
  (LC_ALL/LC_MESSAGES/LANG, the Windows culture, the macOS locale).
  API clients send no language, so SearXNG's "auto" meant "all". Only
  tags the pinned image accepts are written; anything else is "en",
  because an unknown default_lang stops SearXNG from starting.
- Seznam stays enabled, but unless the locale is Czech a hostnames
  rule drops .cz results.
- The factory template gets "en" and the .cz rule.
- tests/test-searxng-config-contract.sh checks the locale mapping,
  renders the Linux, macOS and Windows generators and requires identical
  output, ties the tag list to the image pin, and is now run by CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/searxng-locale.sh
ods/installers/macos/lib/env-generator.sh
ods/installers/phases/06-directories.sh
ods/installers/windows/lib/env-generator.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs
Lightheartdevs merged commit 8c589ef into main Sep 26, 2026
53 checks passed
@Lightheartdevs
Lightheartdevs deleted the claude/searxng-locale branch October 4, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant