Skip to content

fix(cli): stop reporting "up to date" when the Homebrew update check fails - #1298

Open
groksrc wants to merge 3 commits into
mainfrom
fix/1297-brew-update-false-uptodate
Open

fix(cli): stop reporting "up to date" when the Homebrew update check fails#1298
groksrc wants to merge 3 commits into
mainfrom
fix/1297-brew-update-false-uptodate

Conversation

@groksrc

@groksrc groksrc commented Aug 23, 2026

Copy link
Copy Markdown
Member

Closes #1297

Problem

_check_homebrew_update_available() decided outdated-or-not purely from brew outdated's stdout and discarded result.returncode:

stdout = (result.stdout or "").strip()
is_outdated = PACKAGE_NAME in stdout
return is_outdated, None

Every failure mode of brew outdated produces the same shape — non-zero exit, empty stdout, reason on stderr — so a failed check read as "not outdated" and run_auto_update() fell into its if not update_available branch and reported UP_TO_DATE. A hard failure to determine the latest version was reported to the user as success, which silently pins a Homebrew install to whatever version it happens to have.

The trigger seen in the wild was newer Homebrew's third-party tap trust requirement:

$ brew outdated --quiet basic-memory
Error: Refusing to load formula basicmachines-co/basic-memory/basic-memory from untrusted tap basicmachines-co/basic-memory.
$ echo $?
1

but a stale tap, a formula brew cannot resolve, brew off PATH, or a network failure all produce the identical empty-stdout shape. The fix targets the general failure mode, not that one message.

Secondary defect on the same path: the function returned latest_version=None unconditionally, so even a correct "outdated" result printed Update available (latest: unknown).

Fix

Model the three real outcomes instead of two:

brew outdated result outcome
exit 0, empty stdout up to date
package name on stdout outdated
anything else (non-zero exit + empty stdout, timeout, FileNotFoundError) unansweredHomebrewCheckError

run_auto_update() catches HomebrewCheckError, logs brew's reason, and falls back to _check_pypi_update_available().

Why PyPI is a sound fallback for a Homebrew install: the tap is updated from a PyPI release, so it can only lag PyPI, never lead it. If PyPI's latest equals the installed version, no newer version exists anywhere and "up to date" is true. source stays HOMEBREW, so _manual_update_hint() still says brew upgrade basic-memory and the auto-install path still runs brew upgrade. As a bonus, when brew itself is the broken part, that upgrade surfaces brew's real error (e.g. the trust message) to the user instead of a silent no-op.

If PyPI is also unreachable, the existing outer handler reports FAILED with the error — still never a false all-clear.

Signalling the unanswered case with an exception keeps this branch symmetric with _check_pypi_update_available(), which already raises when it cannot compare versions, and matches the house rule of reserving exceptions for unpredictable subprocess/network failures.

Verification

Confirmed the false negative against unmodified main using real brew on macOS — _check_homebrew_update_available() returned (False, None) for an exit-1/empty-stdout/Error:-on-stderr result. End to end, same fake result through run_auto_update(check_only=True) with a Homebrew executable path:

before: up_to_date       | Basic Memory is up to date (0.22.1).
after:  update_available | Update available (latest: 9.9.9). Run `brew upgrade basic-memory`.

Four regression tests added to tests/cli/test_auto_update.py:

  • test_check_homebrew_update_available_failed_check_is_not_up_to_date — the untrusted-tap shape (exit 1, empty stdout, stderr) raises instead of returning "not outdated"
  • test_check_homebrew_update_available_reports_missing_brewFileNotFoundError for brew is an unanswered check
  • test_failed_homebrew_check_falls_back_to_pypi — end to end: a failed brew check yields UPDATE_AVAILABLE with a real latest_version and a brew upgrade hint, not UP_TO_DATE
  • test_failed_homebrew_check_reports_failure_when_pypi_is_unreachable — with neither source able to answer, FAILED
uv run pytest tests/cli/test_auto_update.py
24 passed in 1.79s

uv run ruff check / ruff format --check clean on both files; uv run ty check src tests test-int reports the same 4 pre-existing pymilvus unresolved-import diagnostics as clean main — no new lint or type findings.

Note: tests/cli/ has pre-existing failures on clean main in this environment (ANSI-colored console output vs. plain-text assertions, count varies run to run under pytest-randomly). They are unrelated to this change and unaffected by it.

`_check_homebrew_update_available()` decided outdated-or-not purely from
`brew outdated`'s stdout and discarded the return code. Every failure mode
of that command produces the same shape -- non-zero exit, empty stdout,
reason on stderr -- so a failed check was read as "not outdated" and the
caller reported `UP_TO_DATE`. A hard failure to determine the latest
version was reported to the user as success, silently pinning Homebrew
installs to whatever version they happened to have.

Observed with newer Homebrew's third-party tap trust requirement
("Refusing to load formula ... from untrusted tap"), but a stale tap, a
missing formula, brew off PATH, or a network failure all produce the same
empty-stdout shape.

Model the three real outcomes instead of two: exit 0 with empty stdout is
up to date, the package name on stdout is outdated, and anything else is
unanswered and now raises `HomebrewCheckError`. `run_auto_update()` catches
it and falls back to the PyPI comparison, which can answer the question --
the tap can only lag PyPI, never lead it -- and yields a real
`latest_version`, so Homebrew users no longer see "Update available
(latest: unknown)". `source` is unchanged, so remediation still points at
`brew upgrade basic-memory`. If PyPI is also unreachable the existing
handler reports FAILED rather than a false all-clear.

Closes #1297

Signed-off-by: Drew Cain <groksrc@gmail.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2f66fb7bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# comparison is a sound answer, and it yields a real latest version.
# `source` is unchanged, so remediation still points at `brew upgrade`.
logger.warning(f"Homebrew update check failed, falling back to PyPI: {exc}")
update_available, latest_version = _check_pypi_update_available()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't auto-upgrade Homebrew based on PyPI state

When brew outdated fails while PyPI has published a newer release but the Homebrew tap has not caught up, this fallback sets update_available=True and the normal periodic/MCP path proceeds to brew upgrade even though that version is unavailable through Homebrew. This window is explicitly possible because .github/workflows/release.yml publishes to PyPI in the release job before the dependent Homebrew formula job runs; a Homebrew-check failure during that window therefore causes a misleading availability result and a spurious automatic upgrade attempt. Preserve an unanswered/failed Homebrew outcome, or prevent automatic installation when availability was inferred only from PyPI.

AGENTS.md reference: AGENTS.md:L146-L148

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — confirmed and fixed in 2f3aa9b.

Verified both halves of the claim before acting:

  • release.yml: the homebrew job declares needs: release, and release publishes to PyPI. So the window where PyPI leads the tap is real, not theoretical.
  • Control flow: unless check_only or source == UNKNOWN, the function does fall through to --- Automatic install --- and run brew upgrade.

There is a second reason the old behaviour was wrong, beyond the release window: whatever prevented brew outdated from answering (untrusted tap, brew missing) also blocks brew upgrade, so the auto-install was doomed regardless of tap lag. The new regression test demonstrates exactly that — without the fix it does not merely mis-report, it returns FAILED because the upgrade actually ran and errored.

The fix keeps the asymmetry rather than dropping the fallback: PyPI stays authoritative for the negative answer, since the tap can only lag PyPI and never lead it, so "nothing newer exists" is still sound. Only the positive answer is unsafe to act on. When availability was inferred from PyPI because brew could not answer, the result is now UPDATE_AVAILABLE with updated=False, surfacing the brew failure reason plus the brew upgrade hint, and no subprocess is launched.

Two tests added: one asserting no ["brew", "upgrade", ...] call on that path, one pinning the negative-answer behaviour so a future change does not "fix" the asymmetry away.

On retry/backoff, which was also suggested: deliberately not doing it. The dominant failure modes here are persistent, not transient — an untrusted tap is a policy gate and a missing brew will not heal on a second attempt — so backoff would add latency to an interactive command and to the silent MCP path while only reducing how often the fallback is reached, not making it correct.

`brew outdated` writes progress output ("==> Downloading Homebrew API
data") to stderr on the successful outdated path, so a non-empty stderr is
not an error signal on its own. The outdated-case test asserted against an
empty stderr, leaving nothing to catch a future "stderr means failure"
reading. Use brew's real stderr instead.

This locks the three real result shapes into the suite: exit 0 with empty
stdout (up to date), exit 1 with the tap-qualified formula name on stdout
(outdated), and exit 1 with empty stdout plus an `Error:` stderr
(unanswered). Exit 1 is shared by the middle and last, so stdout stays the
discriminator and the return code only breaks the tie when stdout is empty.

Signed-off-by: Drew Cain <groksrc@gmail.com>
@groksrc

groksrc commented Aug 23, 2026

Copy link
Copy Markdown
Member Author

Follow-up (e136bad) pinning one more empirically-confirmed detail into the tests.

brew outdated --quiet basic-memory exits 1 in both the outdated case and the failure case, so the return code alone cannot discriminate — and on the successful outdated path brew also writes to stderr:

exit 1 | stdout: "basicmachines-co/basic-memory/basic-memory" | stderr: "==> Downloading Homebrew API data"
exit 1 | stdout: ""                                           | stderr: "Error: Refusing to load formula ... from untrusted tap ..."

So neither "non-zero exit" nor "non-empty stderr" is an error signal on its own. The fix keys on stdout first and only consults the return code to break the tie when stdout is empty; stderr is used solely as message text after the decision is made. Two consequences worth flagging for review:

  • The formula name on stdout is tap-qualified (basicmachines-co/basic-memory/basic-memory), not the bare basic-memory. The substring check in _check_homebrew_update_available() is load-bearing — tightening it to an exact match against PACKAGE_NAME would break the outdated path.
  • The outdated-case test previously asserted against an empty stderr. It now carries brew's real stderr chatter, so a future "stderr means failure" reading fails the test rather than silently regressing.

Verified the suite rejects the naive implementation: replacing the body with if returncode != 0: raise fails test_check_homebrew_update_available_exit_code_1_means_outdated. All three shapes — (0, empty), (1, name on stdout), (1, empty + Error: stderr) — are now covered, with only the third refusing to report "up to date". 24 passed.

The PyPI fallback for a failed `brew outdated` set update_available=True,
and run_auto_update() then proceeded to run `brew upgrade`. That is unsafe
two ways: release.yml publishes to PyPI in the `release` job while the
Homebrew formula job `needs: release`, so PyPI can carry a version the tap
cannot install yet; and whatever hid the brew answer (untrusted tap, brew
missing) also blocks the upgrade itself, so the command is doomed.

Keep the fallback for the negative answer -- the tap can only lag PyPI,
never lead it, so "nothing newer exists" is sound -- but when availability
was inferred from PyPI because brew could not answer, report the update
plus the brew failure and let the user upgrade deliberately.

Reported by Codex review on #1298.

Signed-off-by: Drew Cain <groksrc@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] bm update reports "up to date" when the Homebrew check fails (false negative pins users to an old version)

1 participant