[platform-modules]: tolerate kernel-headers race during clean#27524
Closed
bhouse-nexthop wants to merge 1 commit into
Closed
[platform-modules]: tolerate kernel-headers race during clean#27524bhouse-nexthop wants to merge 1 commit into
bhouse-nexthop wants to merge 1 commit into
Conversation
Wrap the per-module `make -C $(KERNEL_SRC)/build M=... clean` in
`override_dh_auto_clean` with `|| true` so a concurrent
platform-modules-* recipe's transient uninstall/reinstall of the
linux-headers package can't fail the recipe.
The race:
1. dpkg-buildpackage runs `debian/rules clean` at the start of
every build pass.
2. Each vendor's override_dh_auto_clean iterates over its
MODULE_DIRS (Dell has 18 platforms, Delta has several, etc.)
and runs `make -C /lib/modules/$KVERSION/build M=<path> clean`
per platform. The iteration takes seconds-to-tens-of-seconds.
3. Under parallel builds, another platform-modules-* recipe may
declare the linux-headers .debs as prerequisites and trigger
SONiC's dep_install/uninstall flow concurrently. If the
linux-headers package is uninstalled (even momentarily,
between dpkg purge and reinstall) while this iteration is
mid-flight, the next `make -C /lib/modules/$KVERSION/build`
fails fast with "No such file or directory" on the build
symlink, aborting the whole recipe.
The race is real-world reproducible under SONIC_BUILD_JOBS=12 on
master. Observed during a Broadcom build:
make[3]: *** /lib/modules/6.12.41+deb13-sonic-amd64/build: \
No such file or directory. Stop.
make[2]: *** [debian/rules:101: override_dh_auto_clean] Error 2
[...]
make: *** [slave.mk:915: target/debs/trixie/\
platform-modules-z9100_1.1_amd64.deb] Error 1
The clean step's job is to remove stale build artifacts from the
in-tree module source dirs (Module.symvers, *.o, *.mod.c). If the
kernel build dir is briefly missing and the kernel-make-clean is
skipped, the next `override_dh_auto_build` step rebuilds those
artifacts from scratch anyway — kbuild regenerates them
deterministically. So a missed clean is harmless; an aborted clean
is not.
Patches every vendor `debian/rules` that has the same iteration
pattern:
platform/aspeed/sonic-platform-modules-nexthop
platform/barefoot/sonic-platform-modules-ingrasys
platform/barefoot/sonic-platform-modules-netberg
platform/broadcom/sonic-platform-modules-dell
platform/broadcom/sonic-platform-modules-delta
platform/broadcom/sonic-platform-modules-ingrasys
platform/broadcom/sonic-platform-modules-inventec
platform/broadcom/sonic-platform-modules-mitac
platform/broadcom/sonic-platform-modules-nexthop
platform/centec/sonic-platform-modules-e582
platform/marvell-teralynx/sonic-platform-modules-cel
platform/marvell-teralynx/sonic-platform-modules-delta
platform/marvell-teralynx/sonic-platform-modules-netberg
platform/marvell-teralynx/sonic-platform-modules-supermicro
platform/nephos/sonic-platform-modules-accton
platform/nephos/sonic-platform-modules-ingrasys
Two of the patched recipes (aspeed/nexthop, broadcom/nexthop)
already have an outer `if [ -d "$(KERNEL_BUILD)" ] ...` guard, but
the dir can disappear between the check and the make execution —
the `|| true` closes that race window for them too.
Signed-off-by: Brad House <bhouse@nexthop.ai>
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
Author
|
Withdrawing in favor of sonic-net/saibcm-modules#39, which fixes the actual root cause. After more investigation, the kernel-headers symlink wasn't disappearing due to dpkg install/uninstall churn or depmod (depmod doesn't touch the The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
Multiple vendor
sonic-platform-modules-*debian/rules files have anoverride_dh_auto_clean(oroverride_dh_clean) that iterates over a list of per-platform module dirs and runs:make -C \$(KERNEL_SRC)/build M=\$(MOD_SRC_DIR)/\${mod}/modules cleanper platform. The iteration takes seconds (Dell has 18 platforms; others fewer). Under parallel builds (e.g.
SONIC_BUILD_JOBS=12), another concurrent platform-modules-* recipe can declare thelinux-headers-*debs as prerequisites and trigger SONiC's dep_install / dep_uninstall flow. Iflinux-headers-*is momentarily uninstalled (purge-then-reinstall) while this iteration is mid-flight, the nextmake -C /lib/modules/\$KVERSION/build ...fails fast withNo such file or directoryon thebuildsymlink, aborting the entire recipe.Real-world failure observed during a Broadcom build with
SONIC_BUILD_JOBS=12:The clean step's purpose is to remove stale in-tree build artifacts (
Module.symvers,*.o,*.mod.c). If a missed clean leaves those behind, the very nextoverride_dh_auto_buildregenerates them deterministically from kbuild — a missed clean is harmless. An aborted clean, on the other hand, kills the recipe.Work item tracking
How I did it
Wrapped each racy
make -C \$(KERNEL_SRC)/build M=... clean(or\$(KERNEL_BUILD) M=...) with|| trueso a transient missing-or-mid-purge kernel-headers package no longer fails the recipe. One-line diff per vendor file, 16 files total.Two of the affected recipes (aspeed/nexthop, broadcom/nexthop) already have an outer
if [ -d \"\$(KERNEL_BUILD)\" ] ...guard, but the dir can still disappear between that check and the actualmakeexecution — the|| truecloses that race window for them too.Affected files:
How to verify it
Alternatively, reproduce more deterministically by injecting an artificial sleep into one platform's clean step while another concurrently runs `dpkg --remove linux-headers-$KVERSION-sonic-amd64` then `dpkg --install ...`. Without the patch the clean fails; with the patch it does not.
Which release branch to backport (provide reason below if selected)
n/a (master-only fix; the race exists wherever the parallel-build infra runs).
Tested branch (Please provide the tested image version)
Description for the changelog
`platform-modules`: tolerate concurrent-build races against the linux-headers package during the per-platform clean iteration. Prevents intermittent build failures with `make[3]: *** /lib/modules/$KVERSION/build: No such file or directory` under parallel `SONIC_BUILD_JOBS`.
Link to config_db schema for YANG module changes
n/a
A picture of a cute animal (not mandatory but encouraged)
n/a