From c5682a7222763985332e52bfde039c8c53033225 Mon Sep 17 00:00:00 2001 From: VizzleTF Date: Fri, 4 Sep 2026 18:23:09 +0300 Subject: [PATCH] fix(images): refresh the opkg index in the layer that installs `opkg update` ran in an earlier RUN than the install of the staged extra_packages files. Buildkit keeps that layer for as long as the release and the feed package list hold, while the install layer re-runs on every change to the staged files, so the index could be months older than the install reading it. A pin does not make that safe: OpenWrt rebuilds the packages inside releases/24.10.8/ in place, under the same version string. Measured on 2026-09-04 against an image built 2026-07-29: index in the image bash 5.2.37-r1 Size 473650 SHA256 f1872e60... downloads.openwrt.org bash 5.2.37-r1 Size 473647 SHA256 20eaa220... opkg_install_pkg: Checksum or size mismatch for package bash. Either the opkg or the package index are corrupt. Try 'opkg update'. `owlab test` had the same hole at run time. It refreshed the index only when a feed had been added, but a package installed by path still resolves its DEPENDENCIES out of the index, and a reused image carries whatever index its build left behind. The apk line needs neither change, measured: apk-tools 3.0.5 revalidates a cached index older than --cache-max-age (4 hours by default) on its own, and re-downloaded every APKINDEX before resolving anything in a month-old image. opkg has no such policy. Nothing that cached before stops caching. An unchanged `owlab up` is still 28 cached steps in 1.6 s; changing only the staged package leaves the feed-install layer CACHED and costs about 1.6 s and a megabyte of gzipped indexes in the layer that was re-running anyway. --- CHANGELOG.md | 23 ++++++++++++++++ cmd/owlab/test.go | 15 ++++++++++- docs/troubleshooting.md | 52 +++++++++++++++++++++++++++++++++++- docs/troubleshooting_ru.md | 54 +++++++++++++++++++++++++++++++++++++- images/Dockerfile | 29 ++++++++++++++++++++ 5 files changed, 170 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c18a15f..c5c8c03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,29 @@ Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html). today keeps working across minor and patch releases; a change that would break one waits for a major. +## [Unreleased] + +### Fixed + +- **A 24.10 image build no longer fails on a package nobody asked for.** `opkg update` + ran in an earlier layer than the install of the `extra_packages:` files, and buildkit + keeps that layer for as long as the release and the feed package list hold — while the + install layer re-runs on every change to the staged files. The index the install read + could therefore be months old, and a pin does not make that safe: OpenWrt rebuilds the + packages inside `releases/24.10.8/` in place, under the same version string. Measured + on 2026-09-04 against an image built 2026-07-29, `bash 5.2.37-r1` was 473650 bytes / + `f1872e60…` in the image's index and 473647 bytes / `20eaa220…` on the server, which + opkg reports as `Checksum or size mismatch for package bash` and the build as + `exit code: 255`. `opkg update` now runs in the same layer as the install, and + `owlab test` refreshes the index before installing a local file too — a package + installed by path still resolves its dependencies out of the index. Nothing that + cached before stops caching: the refresh is inside the branch that has something to + install, in the layer that was re-running anyway, and costs about 1.6 s and a megabyte + of gzipped indexes per opkg router. The apk line needs none of this, measured: + apk-tools 3.0.5 revalidates a cached index older than `--cache-max-age` (4 hours by + default) on its own, and re-downloaded every APKINDEX before resolving in a month-old + image. + ## [0.5.5] - 2026-09-04 ### Fixed diff --git a/cmd/owlab/test.go b/cmd/owlab/test.go index b7fd28c..c7d88b6 100644 --- a/cmd/owlab/test.go +++ b/cmd/owlab/test.go @@ -426,7 +426,20 @@ func (a *app) testInstall(ctx context.Context, r *config.Router, run syncpkg.Exe } cmd := pre + pkgmgr.Install(pm, installArgs, pkgmgr.Options{ - Update: len(feed) > 0 || feedSrc != nil, + // Unconditional, not just when a feed was added. A package installed + // by path still resolves its DEPENDENCIES out of the index, and the + // index a router boots with is as old as the image layer that ran + // `opkg update` — which is cached, and on a reused image is whatever + // the feed looked like the week it was built. OpenWrt rebuilds the + // packages inside a pinned point release without bumping a version, + // so an index that old names bytes the server no longer serves: + // + // opkg_install_pkg: Checksum or size mismatch for package bash. + // + // Free on the apk line, which revalidates a cached index older than + // four hours by itself; the cost on the opkg line is one index fetch + // per router per run. + Update: true, // A locally built package carries no signature the router's keyring // knows, and there is no key it could carry that would. Untrusted: len(files) > 0, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 0a61d5d..385a78d 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -8,6 +8,48 @@ when you hit them, because almost none of them presents as its cause. --- +## "The image build fails" + +### The index in the image is older than the layer that uses it + +**Symptom.** `owlab up` fails while installing an `extra_packages:` file on a +24.10 router, and names a package nobody asked for: + +``` +opkg_install_pkg: Checksum or size mismatch for package bash. Either the opkg +or the package index are corrupt. Try 'opkg update'. +owlab: build failed: exit status 1 +``` + +`bash` here is a **dependency** of the staged package, not the staged package. +Only the opkg line does this. + +**Cause.** `opkg update` ran in an earlier layer than the install. Buildkit +keeps that layer for as long as the release and the feed package list hold, +while the install layer re-runs on every change to the staged files — so the +index can be months older than the install reading it. + +A pinned point release does not save you, because it is not frozen: OpenWrt +rebuilds the packages inside `releases/24.10.8/` in place, without bumping a +version. Measured on 2026-09-04 against an image built 2026-07-29: + +``` +index in the image bash 5.2.37-r1 Size 473650 SHA256 f1872e60... +downloads.openwrt.org bash 5.2.37-r1 Size 473647 SHA256 20eaa220... +``` + +Same version, different bytes — which is exactly what opkg reports as a +checksum mismatch. + +**Fix.** `opkg update` runs in the same layer as the install, and `owlab test` +refreshes the index before installing a local file too. apk needs neither: +apk-tools 3.0.5 revalidates a cached index older than `--cache-max-age` +(4 hours by default) on its own, so `apk add` in a month-old image +re-downloads every APKINDEX before it resolves anything. opkg has no such +policy — it reads whatever the last `opkg update` left and never asks. + +--- + ## "The router never answers on its published port" ### The container's interface is in no firewall zone @@ -365,7 +407,7 @@ looking at is worth doing before debugging anything else. --- -## Two upstream behaviours worth knowing +## Three upstream behaviours worth knowing ### `kmods` must stay in the feed list @@ -381,3 +423,11 @@ apk records hard pins in `/etc/apk/world` (`base-files=1707~4ccb782af7`). Pointing a 25.12.4 rootfs at the 25.12.5 feed fails every install with `breaks: world[...]`. owlab does not rewrite the feeds at all, which is the simplest way to keep this right. + +### A pinned point release is still a moving target + +The pin fixes which release you install from, not which bytes that release +serves. `releases/24.10.8/packages/` is rebuilt in place — the `Packages` +index for a release cut months ago was last modified yesterday — and a package +can be replaced under the same version string. So an index is only good for as +long as it is fresh, and every place that reads one has to refresh it first. diff --git a/docs/troubleshooting_ru.md b/docs/troubleshooting_ru.md index 8df71f7..27b89ae 100644 --- a/docs/troubleshooting_ru.md +++ b/docs/troubleshooting_ru.md @@ -8,6 +8,50 @@ --- +## «Сборка образа падает» + +### Индекс в образе старше слоя, который им пользуется + +**Симптом.** `owlab up` падает на установке файла из `extra_packages:` на +роутере 24.10 и называет пакет, который никто не просил: + +``` +opkg_install_pkg: Checksum or size mismatch for package bash. Either the opkg +or the package index are corrupt. Try 'opkg update'. +owlab: build failed: exit status 1 +``` + +`bash` здесь — **зависимость** установленного пакета, а не он сам. Происходит +только на линии opkg. + +**Причина.** `opkg update` выполнялся в более раннем слое, чем установка. +Buildkit держит тот слой ровно столько, сколько не меняются релиз и список +пакетов из фида, а слой установки перезапускается при любом изменении +подготовленных файлов — так что индекс бывает на месяцы старше установки, +которая его читает. + +Пин на точечный релиз не спасает: релиз не заморожен. OpenWrt пересобирает +пакеты внутри `releases/24.10.8/` на месте, не меняя версию. Замерено +2026-09-04 против образа, собранного 2026-07-29: + +``` +индекс в образе bash 5.2.37-r1 Size 473650 SHA256 f1872e60... +downloads.openwrt.org bash 5.2.37-r1 Size 473647 SHA256 20eaa220... +``` + +Одна и та же версия, разные байты — ровно то, о чём opkg говорит как о +несовпадении контрольной суммы. + +**Что сделано.** `opkg update` выполняется в том же слое, что и установка, а +`owlab test` обновляет индекс и перед установкой локального файла. apk не +нужно ни то, ни другое: apk-tools 3.0.5 сам перепроверяет кэшированный индекс +старше `--cache-max-age` (по умолчанию 4 часа), поэтому `apk add` в +месячном образе перекачивает все APKINDEX прежде, чем что-либо разрешать. У +opkg такой политики нет — он читает то, что оставил последний `opkg update`, и +ничего не спрашивает. + +--- + ## «Роутер вообще не отвечает на опубликованном порту» ### Интерфейс контейнера не попал ни в одну зону firewall @@ -360,7 +404,7 @@ VM не грузится или виснет на середине, тогда --- -## Два поведения апстрима, которые стоит знать +## Три поведения апстрима, которые стоит знать ### `kmods` должен остаться в списке фидов @@ -376,3 +420,11 @@ apk записывает жёсткие пины в `/etc/apk/world` (`base-file Направив rootfs 25.12.4 на фид 25.12.5, вы получите падение любой установки с `breaks: world[...]`. owlab не переписывает фиды вообще — самый простой способ не сломать это. + +### Пин на точечный релиз всё равно движется + +Пин фиксирует, из какого релиза вы ставите, но не то, какие байты этот релиз +отдаёт. `releases/24.10.8/packages/` пересобирается на месте — индекс +`Packages` релиза, вышедшего месяцы назад, изменён вчера, — и пакет может быть +заменён под той же строкой версии. Значит, индекс годен ровно пока он свежий, и +каждое место, которое его читает, обязано сначала его обновить. diff --git a/images/Dockerfile b/images/Dockerfile index 2ea35ef..7ee405b 100644 --- a/images/Dockerfile +++ b/images/Dockerfile @@ -135,6 +135,34 @@ COPY extra/ /tmp/owlab-extra/ # installing one at a time makes that a hard error ("unable to select # packages") for anything but the luckiest ordering. Handed the whole set, the # package manager resolves among them. +# +# opkg's index is refreshed HERE, in the layer that uses it, and that is the +# whole reason the line exists. The `opkg update` above runs in a different +# RUN, whose inputs are the release and the feed package list; buildkit keeps +# that layer for as long as those hold, while this one re-runs on every change +# to the staged files. The two are routinely months apart. +# +# That is fatal because a pinned point release is not frozen. OpenWrt rebuilds +# the packages inside `releases/24.10.8/` in place, without bumping a version, +# so an index from an older build names bytes the server no longer serves: +# +# image built 2026-07-29 bash 5.2.37-r1 Size 473650 SHA256 f1872e60... +# downloads.openwrt.org bash 5.2.37-r1 Size 473647 SHA256 20eaa220... +# +# opkg_install_pkg: Checksum or size mismatch for package bash. Either the +# opkg or the package index are corrupt. Try 'opkg update'. +# +# It bites here rather than above because a local package is installed by +# path, but its DEPENDENCIES still come out of the index. +# +# apk gets no equivalent line, measured: apk-tools 3.0.5 revalidates a cached +# index older than --cache-max-age (4 hours by default) on its own, so an +# `apk add` inside a month-old image re-downloads every APKINDEX before it +# resolves anything, and a second one seconds later does not. opkg has no such +# policy — it reads whatever the last `opkg update` left and never asks. +# +# Inside the `if`, so a build with no out-of-feed package fetches nothing, and +# in this layer only, so nothing that cached before still caches. RUN set -eu; \ dir="/tmp/owlab-extra/$ROUTER_ID"; \ set -- ; \ @@ -144,6 +172,7 @@ RUN set -eu; \ if [ "$PKG_MANAGER" = "apk" ]; then \ apk add --allow-untrusted --force-overwrite "$@"; \ else \ + opkg update; \ opkg install --force-overwrite "$@"; \ fi; \ fi; \