utils/curl: guess charset from a <meta> tag

[hyuraku]

curl_http_content_headers_and_checksum reads the page body to compare the HTTP and HTTPS versions of a homepage. It only took the charset from the HTTP header, treating pages without one as UTF-8. #13223 left a TODO to also read it from <meta> tags, after a page like https://opencsg.org/ that only declares it there.

Now it also detects charsets from <meta> tags within the first 1024 bytes, supporting both charset and http-equiv forms. The HTTP header still takes priority.

It also ignores non-ASCII-compatible encodings like UTF-16 to prevent Encoding::CompatibilityError during regex processing.

Added tests for charset detection, header priority, the 1024-byte limit, and UTF-16 handling.

---

• Have you followed our Contributing guidelines?
• Have you checked for other open Pull Requests for the same change?
• Have you explained what your changes do? Performance claims (e.g. "this is faster") must include brew benchmark results.
• Have you explained why you'd like these changes included, not just what they do?
• For bug fixes, have you given step-by-step brew commands to reproduce the bug?
• Have you written new tests (excluding integration tests)? Here's an example.
• Have you successfully run brew lgtm (style, typechecking and tests) locally?

---

• I did not use AI/LLM to create this PR, or I disclosed the tool/model below and reviewed its output; I did not attribute commits to AI and will answer maintainer questions and review comments myself without AI/LLM.

AI (Claude Code) helped me pick this TODO and wrote the code and tests. I reviewed the change and ran brew lgtm locally.

---

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of downloaded content encodings by recognizing charset declarations in the Content-Type header or, when absent, HTML metadata within the first 1,024 bytes. HTML metadata inside comments is ignored.
  • Header declarations take precedence over HTML metadata. Unknown or non-ASCII-compatible encodings are not applied, so content uses the default external encoding.
  • Content checks now compare responses consistently across different charsets and handle content that cannot be converted to UTF-8.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 03155bf3-d9c0-4745-824e-bedda3fbeeac

📥 Commits

Reviewing files that changed from the base of the PR and between f6ca5f6 and dab4105.

📒 Files selected for processing: 2

• Library/Homebrew/test/utils/curl_spec.rb
• Library/Homebrew/utils/curl.rb

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

The curl utilities now detect response charsets from headers and HTML meta tags. The HTTP-to-HTTPS content comparison converts response bodies to UTF-8 when possible. Tests cover charset detection, fallback behaviour, and comparison across encodings.

Changes: Response encoding

Layer / File(s)	Summary
Detect and apply response encoding Library/Homebrew/utils/curl.rb, Library/Homebrew/test/utils/curl_spec.rb	The reader checks the Content-Type charset before checking HTML meta tags in the first 1,024 bytes. It accepts quoted charset values and applies recognized encodings only when they are ASCII-compatible. Tests cover precedence and fallback cases.
Normalise bodies before comparison Library/Homebrew/utils/curl.rb, Library/Homebrew/test/utils/curl_spec.rb	The comparison utility normalises both response bodies and converts them to UTF-8 with invalid or undefined characters replaced. If Ruby has no converter for a body’s encoding, it retains that body’s original encoding. Tests cover equivalent content in different charsets and unavailable converters.

Priority: ➖ Normal

Merge Risk: 🔵 Low · up to dab41

Some pages can still be read with the wrong encoding, making HTTP and HTTPS content comparisons inaccurate. Resolve the remaining meta-parsing issue or accept this bounded audit risk before merging.

Security architecture risk: ⚪ Minimal · up to dab41

The change affects text interpretation for existing URL audits, not download integrity or access privileges. Checksums remain calculated from the original downloaded bytes, and existing HTTPS and mirror-validation controls remain unchanged. No material security risk introduced or worsened by this PR was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — A remote response source, including an attacker able to modify plain-HTTP traffic, can influence decoding and resulting HTTPS audit recommendations. The inspected propagation reaches existing audit outcomes, not a new credential, privilege, or checksum authority. Response content and headers already influenced those outcomes before this PR.

Trust Boundaries and Controls

• observed — Response metadata remains untrusted input to text interpretation. A captured header charset takes precedence; the fallback is byte-bounded and comment-stripped, and only recognized ASCII-compatible encodings are applied. The regex does not enforce full HTML attribute semantics or an http-equiv requirement, so its result is a heuristic rather than browser-equivalent parsing.
• observed — Decoded equivalence is separate from byte integrity. Existing status and redirect checks precede comparison, and a normalized match returns an HTTPS recommendation rather than validating an artifact checksum.

Resilience and Maintainability Implications

• observed — The changed decoding path adds reads and local string conversion, not persistent shared state or file rewriting. Unknown charsets retain the default reader encoding, unavailable converters retain the scrubbed content, and the existing ensure block unlinks the per-call temporary file during normal return or exception unwinding.

Pre-merge checks: 4

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarises the main change: detecting a charset from a tag in utils/curl. It is concise and specific.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 68d7f5f1-0805-4109-8f79-54cc9f2bdff1

📥 Commits

Reviewing files that changed from the base of the PR and between 92b8ea8 and 07254bc.

📒 Files selected for processing: 2

• Library/Homebrew/test/utils/curl_spec.rb
• Library/Homebrew/utils/curl.rb

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout: 19

• GitHub Check: upload_sarif
• GitHub Check: docker (x86_64 Ubuntu 26.04)
• GitHub Check: docker (x86_64 Ubuntu 24.04)
• GitHub Check: docker (arm64 Ubuntu 26.04)
• GitHub Check: docker (arm64 Ubuntu 24.04)
• GitHub Check: tests (Linux, 1/2)
• GitHub Check: tests (load-only macOS)
• GitHub Check: tests (macOS, 2/2)
• GitHub Check: tests (online, 1/2)
• GitHub Check: tests (online, 2/2)
• GitHub Check: tests (no Sorbet)
• GitHub Check: tests (generic OS, 1/2)
• GitHub Check: tests (generic OS, 2/2)
• GitHub Check: tests (macOS, 1/2)
• GitHub Check: tests (Linux, 2/2)
• GitHub Check: tests (load-only Linux)
• GitHub Check: Analyze
• GitHub Check: syntax
• GitHub Check: docs

🔇 Additional comments: 1

Library/Homebrew/test/utils/curl_spec.rb: 1

708-744: LGTM!

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: c08c252e-f87a-45a8-b364-336c07927b3d

📥 Commits

Reviewing files that changed from the base of the PR and between 07254bc and a8c223e.

📒 Files selected for processing: 2

• Library/Homebrew/test/utils/curl_spec.rb
• Library/Homebrew/utils/curl.rb

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout: 19

• GitHub Check: docker (arm64 Ubuntu 24.04)
• GitHub Check: docker (x86_64 Ubuntu 26.04)
• GitHub Check: docker (arm64 Ubuntu 26.04)
• GitHub Check: docker (x86_64 Ubuntu 24.04)
• GitHub Check: workflow_syntax
• GitHub Check: tests (load-only macOS)
• GitHub Check: tests (no Sorbet)
• GitHub Check: tests (Linux, 2/2)
• GitHub Check: tests (Linux, 1/2)
• GitHub Check: tests (generic OS, 2/2)
• GitHub Check: tests (online, 1/2)
• GitHub Check: tests (macOS, 2/2)
• GitHub Check: tests (macOS, 1/2)
• GitHub Check: tests (load-only Linux)
• GitHub Check: tests (online, 2/2)
• GitHub Check: tests (generic OS, 1/2)
• GitHub Check: docs
• GitHub Check: syntax
• GitHub Check: Analyze

[MikeMcQuaid]

Good catch, one wee comment!

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

🟡 Minor · Normalize decoded bodies before comparison. · curl.rb:681-691

Library/Homebrew/utils/curl.rb:681-691
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Normalize decoded bodies before comparison.

When HTTP and HTTPS responses use different ASCII-compatible charsets, scrub and gsub preserve those encodings. Equivalent non-ASCII content can therefore compare unequal. Normalize both bodies to UTF-8 before http_content == https_content.

Suggested fix

-      http_content = details[:file]&.scrub&.gsub(no_protocol_file_contents, "/")
-      https_content = secure_details[:file]&.scrub&.gsub(no_protocol_file_contents, "/")
+      http_content = details[:file]&.scrub&.gsub(no_protocol_file_contents, "/")&.encode(Encoding::UTF_8,
+                                                                                          invalid: :replace, undef: :replace)
+      https_content = secure_details[:file]&.scrub&.gsub(no_protocol_file_contents, "/")&.encode(Encoding::UTF_8,
+                                                                                                invalid: :replace, undef: :replace)

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f01bb3fd-19ad-4e7f-97e9-c3ec73cd52c8

📥 Commits

Reviewing files that changed from the base of the PR and between a8c223e and f6ca5f6.

📒 Files selected for processing: 2

• Library/Homebrew/test/utils/curl_spec.rb
• Library/Homebrew/utils/curl.rb

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout: 17

• GitHub Check: docker (x86_64 Ubuntu 24.04)
• GitHub Check: docker (arm64 Ubuntu 26.04)
• GitHub Check: docker (x86_64 Ubuntu 26.04)
• GitHub Check: docker (arm64 Ubuntu 24.04)
• GitHub Check: docs
• GitHub Check: tests (Linux, 2/2)
• GitHub Check: tests (no Sorbet)
• GitHub Check: tests (macOS, 2/2)
• GitHub Check: tests (online, 2/2)
• GitHub Check: tests (macOS, 1/2)
• GitHub Check: tests (load-only macOS)
• GitHub Check: tests (Linux, 1/2)
• GitHub Check: tests (load-only Linux)
• GitHub Check: tests (generic OS, 2/2)
• GitHub Check: tests (online, 1/2)
• GitHub Check: tests (generic OS, 1/2)
• GitHub Check: syntax

🔇 Additional comments: 2

Library/Homebrew/utils/curl.rb: 1

676-689: LGTM!

Library/Homebrew/test/utils/curl_spec.rb: 1

707-756: LGTM!

[MikeMcQuaid]

@hyuraku above comment looks worth a look.

[MikeMcQuaid]

Thanks!