Skip to content

Proposal: Handle or warn about deprecated sysupdate features to prevent silent update stalls (e.g. Homebrew transition in base image) #688

Open

Description

@asormanni

Hi team! 👋

First of all, thank you so much for the amazing work on Bluefin Server. The DDI + sysupdate architecture is a pleasure to follow and
test.

I'm opening this issue to share a real-world upgrade experience between releases 26.11.2232 and 26.11.2298 and to propose a small UX/migration improvement for future feature deprecations or transitions.

──────

What happened

  1. Starting state: My node was running 26.11.2232, configured with the Home Server profile where the homebrew feature was enabled via:
    /etc/sysupdate.d/homebrew.feature.d/
    At that time, /usr/lib/sysupdate.d/ shipped 39-homebrew-integration.transfer and 40-homebrew.transfer.
  2. The update attempt: When release 26.11.2298 landed, running sudo systemd-sysupdate update silently reported:
    Determining installed update sets…
    Determining available update sets…
    No update needed.
    Even though 26.11.2298 was published as latest and verified successfully by GPG.
  3. Root cause:
    Homebrew transitioned from an opt-in sysext to a first-boot seed in /usr/share/homebrew/brew.tar.zst, dropping the homebrew
    transfer definitions and assets from the release manifest.
    Because systemd-sysupdate requires all enabled features in /etc/sysupdate.d/*.feature.d/ to be satisfied by the remote manifest to consider an update set valid, it silently discarded the candidate 26.11.2298 without any visible error or warning in the standard
    output.
  4. Resolution:
    Manually removing /etc/sysupdate.d/homebrew.feature.d immediately allowed systemd-sysupdate to detect 26.11.2298 as a candidate and successfully complete the update.
    ──────

Why this can be tricky for operators

Unless an operator runs SYSTEMD_LOG_LEVEL=debug systemd-sysupdate list or dives into recent PR diffs, it's not obvious why a newer release is rejected as "No update needed". In an automated setting, the node would simply stop updating without alerting the operator.
──────

Proposal / Ideas

Would it be possible to add a migration helper or check to make transitions like this smoother? Some ideas:

  1. Migration helper in bluefin-host / bluefin-update-status-migrate:
    A one-shot migration service (similar to bluefin-update-status-migrate.service) that detects when an enabled feature in
    /etc/sysupdate.d/ no longer exists in /usr/lib/sysupdate.d/ (or has been deprecated/absorbed into the base image) and automatically archives or removes the stale /etc drop-in.
  2. Diagnostic warning in bluefin-update-status / login banner:
    Have bluefin-update-status or the update check inspect active /etc/sysupdate.d/*.feature.d folders against /usr/lib/sysupdate.d/ and log a noticeable warning (or show it on the login banner) if an orphaned feature is active and blocking updates.
  3. Release notes callout:
    A brief mention in release notes when a feature/transfer is retired from the release manifest, noting that nodes with the feature
    enabled in /etc should remove their drop-ins.

Thanks again for all the incredible work you're putting into this project! Let me know what you think or if I can help test anything
related to this.

Regards,
A.

Activity

  1. added
    needs-kindMissing required label matching ^kind/
    needs-triageMissing required label matching ^triage/
    on Nov 17, 2026
  2. jeefy commented on Nov 17, 2026

    @jeefy
    Collaborator

    Issue triage

    Next action: a triager or maintainer reviews this issue.

    Reporter or contributor

    • If no kind/* label is set, categorize the issue with one of: /kind bug, /kind regression, /kind security, /kind feature, /kind documentation or /kind cleanup.
    • Optionally add an area with /area <area> if this repository supports area labels; use a value from its label list.

    Triager or maintainer

    • Review the report and ask the reporter for any missing information.
    • When the scope is accepted and the issue is ready for work, comment /triage accepted. Only triagers and maintainers can do this; acceptance does not assign an implementer.

    needs-kind clears when a kind is set. needs-triage and this post clear when triage is accepted.

    See Triage & Prow Commands for all available commands.

  3. asormanni commented on Nov 17, 2026

    @asormanni
    Author

    /kind feature

  4. added and removed
    needs-kindMissing required label matching ^kind/
    on Nov 17, 2026
  5. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    Thanks — this hive only works issues from OWNER, MEMBER, COLLABORATOR automatically. A maintainer can admit this one by adding the label triage/accepted (configured in issue_filter.reporter_trust.untrusted_require_labels). Until then the hive will not claim, label, or open PRs for it. See https://github.com/projectbluefin/server/blob/HEAD/GOVERNANCE.md#reporter-trust-and-escalation for what these terms mean and how to become a trusted reporter.

  6. added
    triage/acceptedScope accepted; ready for work. Use `/triage accepted`.
    kind/bugSomething is broken.
    and removed on Nov 17, 2026
  7. Danathar commented on Nov 17, 2026

    @Danathar
    Contributor

    /kind bug

    Thanks, @asormanni, this is a clear write-up and it checks out against the repo. #639 moved Homebrew into the base image as a first-boot seed and removed its sysupdate transfers, but nothing removes the /etc/sysupdate.d/homebrew.feature.d/ drop-in on nodes that had it enabled. So those nodes keep requiring a feature the new release no longer ships, and systemd-sysupdate reports "No update needed". I'm labelling it a bug (an update silently stops), and accepting it.

    Scope:

    1. Fix the stall: a one-shot migration that removes or archives a /etc/sysupdate.d/<feature>.feature.d/ drop-in when no /usr/lib/sysupdate.d/ transfer declares that feature any more, logging what it did. It should run before the update timer.
    2. Make the next one visible: have the update status check report "enabled feature X is not in the release manifest" instead of saying nothing.

    Out of scope: changing how features are declared in the release manifest.

  8. removed
    needs-triageMissing required label matching ^triage/
    on Nov 17, 2026
  9. added
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/015-triage-neededActionable but held by the reporter-trust gate: awaiting a maintainer's triage label
    stage/010-actionableTriaged; eligible for the hive to pick up
    on Nov 17, 2026
  10. hivecommons-hive commented on Nov 18, 2026

    @hivecommons-hive
    Contributor

    Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

    PR #690 covers this issue: #690

    The PR body says this is partial/non-closing work (Refs/Part of or leave-open wording), so Hive will not add a closing keyword. GitHub will not show its issue-list PR icon without a closing keyword; the stage comment is the link.

  11. added a commit that references this issue on Nov 18, 2026
    db69895
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions