Repository navigation
Conversation
scan_rawhide_state.query() unpacked the first non-empty repoquery line into four tab-separated fields, so any dnf5 warning or unexpected line raised ValueError and took down the whole nightly report (issue projectbluefin#172). Skip lines that are not the expected four-field shape, pick the first line that is, and log the discarded line to stderr so a systematically malformed query stays visible. Add unit tests covering a stdout that mixes a warning line with a good one and one with no good line. Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
There was a problem hiding this comment.
correctness
- (low, non-blocking) The loop returns the first line with ≥3 tabs (
tools/scan_rawhide_state.py:35-39). A warning/progress line that happens to contain 3+ tabs and precedes the real repoquery line would be accepted as data and written intorawhide-state.json, feedingsource_name()a garbage sourcerpm (main(), scan_rawhide_state.py:57-62). If dnf5 ever emits tab-formatted noise, consider also validating thatsourcerpmends in.src.rpm. Not a blocker for this fix.
No findings from: security, intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
A warning/progress line can carry 3+ tabs and pass the field-count guard, then feed source_name() a non-SRPM sourcerpm that would raise. Require the sourcerpm to end in .src.rpm in query() so such a line is skipped (this PR's job) instead of reaching source_name(). Regression test added. Addresses the low finding in projectbluefin#206.
@(cat /tmp/c206.md)🐝 Hive Agent: |
There was a problem hiding this comment.
correctness
The .src.rpm suffix guard is weaker than the parser it protects. tools/scan_rawhide_state.py:44 accepts any 4-field line whose last field ends in .src.rpm, but source_name() (tools/rawhide_sources.py:28,40) requires the full SRPM_NAME pattern ^(.+)-[0-9][^-]*-.*\.src\.rpm$ and raises ValueError otherwise. A malformed line like foo\tbar\tbaz\tqux.src.rpm passes the guard, is stored in state, and crashes main() at tools/scan_rawhide_state.py:70 (source_name(value["sourcerpm"])) — verified: source_name('qux.src.rpm') raises. This contradicts the code comment at tools/scan_rawhide_state.py:38-42 ("garbage is skipped here … instead of reaching source_name(), which would raise") and leaves a sliver of the exact crash class issue #172 is about. Suggest validating with SRPM_NAME.match(sourcerpm) instead of endswith(".src.rpm") — the module already imports from tools.rawhide_sources.
Otherwise the fix and tests are sound (all 13 tests pass at e0f5193).
No findings from: security, intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
The guard added in the previous commit only required the fourth field to end in `.src.rpm`. That is a weaker grammar than `SRPM_NAME`, which `source_name()` applies: a line such as `foo\tbar\tbaz\tqux.src.rpm` passed the guard, was written into `config/rawhide-state.json`, and then crashed `main()` when `source_name()` raised `ValueError` on it. That is the residual sliver of the issue projectbluefin#172 crash class the guard was meant to close, and it made the accompanying comment inaccurate. Validate with `SRPM_NAME` so the skip decision uses exactly the grammar that `source_name()` will later demand, and nothing that would raise can reach state. Tests cover the suffix-only case, and a well-formed source RPM whose name contains `-`, so the stronger guard is pinned in both directions. Assisted-by: Claude Opus 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Reviewed: Correctly fixes crash in tools/scan_rawhide_state.py caused by dnf5 warnings or progress output in repoquery stdout by skipping unparseable lines and verifying the sourcerpm field with SRPM_NAME. Adds unit tests and skill documentation. Clean trial merge and passes all tests. Awaiting second independent review to satisfy merge policy.
Danathar
left a comment
There was a problem hiding this comment.
This is a fix for the nightly scan crash in #172, and skipping a genuinely odd line instead of crashing is fine on its own. But I think it targets the wrong cause, and as written it would turn the red nightly into a green nightly that reports nothing.
What actually broke on 09-14. #99 (merged 09-13) changed the parser from split("\\t", 3) to split("\t", 3) on the theory that dnf returns real tabs. dnf5 does not. Its --queryformat only expands \n; a \t is copied through as the two characters backslash and t (libdnf5-cli/output/repoqueryformat.cpp handles only the backslash-n pair). I reproduced it with dnf5 5.4.5.0 in both fedora:45 and fedora:rawhide after the workflow's own dnf -y install dnf-plugins-core python3:
$ dnf -q repoquery --latest-limit=1 --qf "%{name}\t%{evr}\t%{arch}\t%{sourcerpm}" ModemManager
ModemManager\t1.24.2-5.fc45\ti686\tModemManager-1.24.2-5.fc45.src.rpmModemManager\t1.24.2-5.fc45\tx86_64\tModemManager-1.24.2-5.fc45.src.rpm
One line, no real tabs, and the i686 and x86_64 records run together because the format has no newline either. That is the "expected 4, got 1" in the traceback. It is not a warning on stdout: dnf5 writes "Updating and loading repositories" to stderr.
What this branch does with that. With it checked out, query("ModemManager") against dnf5 returns None and logs "skipping unparseable repoquery line" for exactly that line. After merge every package would be None, main() would write {} to config/rawhide-state.json, print "observed 0 Rawhide packages", and exit 0. That hides #172 rather than closing it.
The last green artifact (run 34740193830, 09-13, before #99) shows the older half of the bug too: 21 of 74 entries have the second arch's record glued into sourcerpm, for example mesa-dri-drivers ends in mesa-26.2.2-6.fc46.src.rpmmesa-dri-drivers\t26.2.2-6.fc46\tx86_64\t....
Suggested fix, which I verified gives one clean four-field line per arch with dnf5: pass a real tab and a trailing newline in the format rather than escape sequences, i.e. the Python string "%{name}\t%{evr}\t%{arch}\t%{sourcerpm}\n" with a real tab character. Then pick the x86_64 or noarch record instead of lines[0], since i686 sorts first. The skip-and-log guard can stay on top of that. The mocked test should feed the shape dnf5 actually produces (literal backslash-t, no newline between records) so this cannot regress a third time, and the repeated-mistakes entry should record that the lesson was "#99 assumed dnf4 --qf semantics", not "dnf writes warnings to stdout".
projectbluefin#99 assumed dnf4 --qf semantics on dnf5: dnf5 expands only \n, not \t, in the query format, so a literal \t glues the i686 and x86_64 records into one line. query() then returned None for every package and main() wrote an empty report that exited green, hiding projectbluefin#172 instead of closing it. Use real tabs and a trailing newline in --qf, prefer the x86_64 then noarch record over lines[0] (i686 sorts first), and keep the SRPM_NAME guard. Pin the real tab and the glued-record case in tests so the regression cannot recur unseen. Closes projectbluefin#206 review finding from Danathar.
|
Thanks, Danathar — you found the real root cause and the current fix was going to paper over it. Pushed the actual fix as What was wrong (your point). #99 assumed dnf4 The fix.
Tests. Added Docs. Updated the repeated-mistakes entry: the lesson is now "#99 assumed dnf4 The mocked tests now feed the shape dnf5 actually produces, so this can't regress a third time. Ready for re-review.🐝 Hive Agent: |
There was a problem hiding this comment.
intent-alignment
- The body claims "Log the discarded line to stderr so a systematically malformed query stays visible" — but only 4-field lines whose sourcerpm fails
SRPM_NAMEare logged (tools/scan_rawhide_state.py:49-54). Lines that fail the field-count check are skipped silently (tools/scan_rawhide_state.py:44-45), so the stated visibility for warning/progress lines is not implemented. docs/skills/repeated-mistakes.md rule 16 states the same logging rule the code doesn't fully follow. - The body says "pick the first line that is [four-field]", but the code selects by arch preference (x86_64, then noarch — tools/scan_rawhide_state.py:41). The body also never mentions the substantive
--qfchange (real tabs + trailing newline, tools/scan_rawhide_state.py:26) that is the actual fix for the glued-record failure; it's only explained in code comments and docs.
correctness
- (low) A package whose repoquery output contains only non-x86_64/non-noarch records (e.g., aarch64-only) now returns
Noneand is silently dropped from state (tools/scan_rawhide_state.py:41-56); the old code acceptedlines[0]regardless of arch. Deliberate per the comment, but it is a behavior change beyond crash-avoidance and there is no log when it happens. - (low)
test_query_skips_srpm_suffixed_line_source_name_cannot_parse(tests/test_scan_rawhide_state.py:146-154) passes for the wrong reason: its line's arch field isbaz, so the arch filter rejects it before theSRPM_NAMEguard it claims to pin ever runs. The guard is genuinely covered bytest_query_skips_four_field_line_with_non_srpm_sourcerpm; this one pins nothing.
No findings from: security, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
The PR body and rule 16 promise that a discarded line is logged to stderr so a systematically malformed query stays visible, but only four-field lines with a non-SRPM sourcerpm were logged: lines failing the field-count check hit a bare continue, and a package whose records are all non-x86_64/noarch was dropped by the arch preference with no trace at all. Parse the lines once into an arch-keyed map, logging both discard reasons, then apply the x86_64-then-noarch preference over that map and log the package when neither arch is present. Behavior for well-formed output is unchanged. Also fix test_query_skips_srpm_suffixed_line_source_name_cannot_parse, which used arch 'baz': the arch preference rejected the line before the SRPM_NAME guard the test claims to pin ever ran. It now uses x86_64 and asserts the guard's own stderr message.
Danathar
left a comment
There was a problem hiding this comment.
Approving at 6048c38. This now fixes the cause I described rather than the symptom: tools/scan_rawhide_state.py passes --qf with a real tab and a trailing newline (the Python literal at line 26 evaluates to real tab characters, and test_query_uses_real_tabs_and_trailing_newline pins that), and query() prefers the x86_64 record, then noarch, instead of lines[0].
I verified it against real dnf5 5.4.5.0 in a fedora:rawhide container: the fixed format emits one four-field line per arch with a newline between them, and running the PR's own query() there returns the x86_64 record for ModemManager and the noarch record for dnf-data. Locally at 6048c38, python3 -m pytest tests/test_scan_rawhide_state.py passes (21 tests) and the full suite passes (414 tests). CI is green on this head.
The two points from the automated review on 8f21715 are addressed in 6048c38: every discarded line is now written to stderr, including lines that fail the four-field check and packages with no x86_64/noarch record, and test_query_skips_srpm_suffixed_line_source_name_cannot_parse now uses arch x86_64 so the SRPM_NAME guard is what rejects the line. The repeated-mistakes entry records the dnf4-versus-dnf5 --qf lesson as asked.
castrojo
left a comment
There was a problem hiding this comment.
Approved (+2): Fixes crash in caused by dnf5 progress/warning output in repoquery stdout. Emits proper tabs in --qf, validates sourcerpm with SRPM_NAME regex, prefers x86_64 then noarch records, and logs discarded lines. Verified locally against origin/main: passes factory contract, validation, and all 423 unit tests.
castrojo
left a comment
There was a problem hiding this comment.
Approved (+2): Fixes crash in tools/scan_rawhide_state.py caused by dnf5 progress/warning output in repoquery stdout. Emits proper tabs in --qf, validates sourcerpm with SRPM_NAME regex, prefers x86_64 then noarch records, and logs discarded lines. Verified locally against origin/main: passes factory contract, validation, and all 423 unit tests.
fix(scan): skip malformed repoquery lines instead of crashing the scan
tools/scan_rawhide_state.py::query()unpacked the first non-empty repoqueryline into four tab-separated fields, so any dnf5 warning, progress line, or a
package whose query returned something unexpected raised
ValueError: not enough values to unpackand took down the whole nightlyreport on a scheduled run (#172, red nightly since
at least 09-19). One odd line lost the report for all ~300 packages.
This is observability only, so a single unparseable line should be skipped and
noted, not fatal.
--qfformat. This is thesubstantive fix: test: add unit coverage for tools/scan_rawhide_state.py #99 assumed dnf4
--qfsemantics, but dnf5 expands only\n, not\t, so a literal\tis copied through as two characters and theper-arch records glue into a single line.
query()then returnedNoneforevery package and
main()wrote an empty report that exited green — hidingthe crash rather than closing it.
four-field line whose sourcerpm does not match
rawhide_sources.SRPM_NAME,the same grammar
source_name()later demands, so nothing that would raisedownstream can reach state.
lines[0], which is i686 (arch sorts first). The source package isarch-independent, but a fixed arch keeps the report deterministic.
sourcerpms alike), and log a package that parses cleanly but has no x86_64 or
noarch record, so an arch-filtered drop or a systematically malformed query
stays visible instead of silently returning
None.one with no good line, the glued-record shape, the real-tab format, the arch
preference and its noarch fallback, and each stderr message.
The learning is written back into
docs/skills/repeated-mistakes.md(rule 16)per this repository's skill-improvement mandate.
— hive: backend=pi model=lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K