fix: Suppress bare .svc hostnames and pin lychee version#25
Conversation
Cluster-local Kubernetes service URLs such as http://keycloak-service.keycloak.svc:8080/realms/ were reported as broken links by the scanner. They matched none of the existing suppression clauses: no .svc.cluster.local suffix, non-numeric so not RFC1918, and they resolve to nothing so lychee --exclude-all-private does not catch them. Extract the inline lychee-parsing/suppression jq program into a standalone scripts/extract-broken-links.sh so it can be unit-tested, then add a bare-.svc suppression clause. The scanner calls the extractor with the same append/error semantics. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
The scanner depends on lychee's default exclusion of URLs inside code blocks (inline spans and fenced blocks). Pin a known-good minimum version and document the dependency so a version change cannot silently reintroduce code-block false positives. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
clawgenti
left a comment
There was a problem hiding this comment.
Refactors bare .svc suppression into a standalone, unit-tested extractor — clean separation of concerns with solid test coverage. All checks pass. Ready for human review.
Reviewed by clawgenti using github:pr-review
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| EXTRACTOR="$SCRIPT_DIR/../scripts/extract-broken-links.sh" | ||
| TMPDIR=$(mktemp -d "/tmp/test-extract-broken-XXXXXX") |
There was a problem hiding this comment.
nit: TMPDIR shadows the well-known $TMPDIR environment variable. Subprocesses called from this test script won't be affected (they run in sub-shells and the trap cleanup is correct), but renaming to something like TEST_TMPDIR avoids any confusion with the system env var.
There was a problem hiding this comment.
Good catch — renamed to TEST_TMPDIR in bc7feab. Scoped to this new test file to keep the change focused; the sibling test-parse-diff-map.sh uses TMPDIR too but I left it out of this PR.
Addresses PR review nit: TMPDIR shadows the well-known $TMPDIR environment variable. Rename the test-local temp dir variable to TEST_TMPDIR for clarity. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
clawgenti
left a comment
There was a problem hiding this comment.
Clean refactor extracting the lychee-parsing logic into a standalone, unit-tested script — the bare .svc suppression fix is correct and both regex filters (line 66 and 67) are non-redundant. All checks pass. Ready for human review.
- The PR body states "17 cases" but
test-extract-broken-links.shcontains 18run_testcalls (Test 5 expands to 6 sub-assertions). Minor doc discrepancy. - The stdin (
-) mode documented in theextract-broken-links.shusage block has no corresponding test intest-extract-broken-links.sh.
Reviewed by clawgenti using github:pr-review
| set -euo pipefail | ||
|
|
||
| # Test harness for extract-broken-links.sh | ||
| # Run: bash automation/tests/test-extract-broken-links.sh |
There was a problem hiding this comment.
nit: The usage comment on line 5 says bash automation/tests/test-extract-broken-links.sh, and the script header documents a stdin mode (cat report.json | extract-broken-links.sh - ...) in the extractor, but there's no test exercising the - (stdin) path. Consider adding one to cover that code path.
There was a problem hiding this comment.
Added a stdin (-) path test in c0eb93a — it pipes a fixture through the extractor and asserts the broken link is emitted.
| # here so it can be unit-tested against synthetic lychee fixtures. | ||
| # | ||
| # Usage: | ||
| # extract-broken-links.sh <lychee-json> <repo-name> <repos-prefix> |
There was a problem hiding this comment.
nit: The usage block documents reading from stdin via -, but test-extract-broken-links.sh has no test for this path. The PR body also counts "17 cases" but there are 18 run_test calls — minor mismatch worth fixing in the PR description.
There was a problem hiding this comment.
Both addressed: stdin (-) path now covered by a test in c0eb93a, and the PR body updated to "18 assertions / 18/18 pass" (the file has 17 run_test calls plus one inline stdin assertion; line 35 is the helper definition, not a call).
The extractor documents a stdin mode (cat report.json | ... - ...) but no test exercised it. Add a case piping a fixture through - and asserting the broken link is emitted. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
clawgenti
left a comment
There was a problem hiding this comment.
Adds a well-structured extract-broken-links.sh extractor with 18 tests covering the new bare .svc suppression and regression cases — clean extraction, correct fix. All checks pass. Ready for human review.
Reviewed by clawgenti using github:pr-review
| "docs/f.md" \ | ||
| "https://real-host.example.org/x" \ | ||
| '{"text":"Connection refused"}' | ||
| run_test "unreachable status token" "$TEST_TMPDIR/unreach.json" '.[0].status' 'unreachable' |
There was a problem hiding this comment.
nit: The unreachable path is covered, but the timeout, dns, and error (catch-all) status normalization paths in extract-broken-links.sh have no corresponding test cases. Consider adding fixtures for e.g. '{"text":"Timeout"}' → "timeout", '{"text":"DNS error"}' → "dns", and a non-matching text → "error" to complete the coverage.
There was a problem hiding this comment.
Added in 91cad02: fixtures for the timeout, dns (both resolve and dns matches), and error catch-all text-status branches, plus a null-status case mapping to unknown. All status-normalization paths are now exercised (23/23 assertions).
Add fixtures for the remaining lychee status-normalization branches in extract-broken-links.sh: text statuses mapping to timeout, dns, and the error catch-all, plus a null status mapping to unknown. Previously only the unreachable and numeric-code paths were exercised. Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Gloire Rubambiza <gloire@ibm.com>
esnible
left a comment
There was a problem hiding this comment.
Clean, well-scoped fix. I fetched the PR head from the fork and verified independently:
- All 23 tests pass locally (
tests/test-extract-broken-links.sh); shellcheck clean on both changed scripts. - The bare-
.svcregex is precise. It suppressesfoo.ns.svc[:port],.svc.cluster.local,.local, and RFC1918 ranges, and correctly passes through hostnames that merely contain.svcas a non-terminal label (foo.svc.example.com) or in the path (/foo.svc) — no over-suppression of real external URLs. - Both
.svcclauses are necessary. The bare-.svcregex does not match the.svc.cluster.localform (the char after.svcis., not:///end), so neither suppression clause is dead code. - stdin (
-), file-path, and usage/missing-file exit codes all behave as documented. - Error semantics preserved. The scanner still wraps the extractor call in
2>/dev/null || true, identical to the original inline jq — no regression.
The refactor into a unit-testable extract-broken-links.sh is faithful, status normalization is exercised across all branches, and the lychee >= 0.23.0 pin is a reasonable defensive note.
Assisted-By: Claude Code
Summary
Fixes two false-positive classes in the link-health scanner reported by @esnible
(rossoctl/cortex#532, #533).
Bare
.svcsuppression gap. Cluster-local Kubernetes URLs such ashttp://keycloak-service.keycloak.svc:8080/realms/were flagged as broken. Theymatched none of the existing suppression clauses (no
.svc.cluster.localsuffix,non-numeric so not RFC1918, resolve to nothing so
--exclude-all-privatemissesthem). Added a bare-
.svcclause.lychee version pin (defensive). Verified lychee 0.23.0 already excludes
URLs inside code blocks by default (
--include-verbatimopts them back in). Pinneda minimum version and documented the dependency so this cannot regress.
Changes
unit-testable
scripts/extract-broken-links.sh; scanner now calls it (sameappend/error semantics).
.svcsuppression clause.tests/test-extract-broken-links.sh(23 assertions): bare-.svcsuppressionplus regressions (
.svc.cluster.local,.local, RFC1918 ranges stay suppressed;genuine internal/external broken links pass through with correct fields), edge
cases (empty/missing
error_map, non-URL entries), and the stdin (-) path.docs/running-without-openclaw.md.Testing
bash tests/test-extract-broken-links.sh— 23/23 pass.bash tests/test-parse-diff-map.sh— 45/45 pass (no collateral breakage)..svcURL outside a codeblock extracts it into
error_map; the extractor drops it (zero false positive).shellcheckclean on both changed scripts.Fixes #23
Fixes #24
Assisted-By: Claude Code