Skip to content

πŸ› fix: avoid empty reader catch-up pulls for filtered lines - #16569

Closed
thatguywob wants to merge 2 commits into
hivecommons:v5from
thatguywob:fix/16530-reader-catch-up
Closed

thatguywob wants to merge 2 commits into
hivecommons:v5from
thatguywob:fix/16530-reader-catch-up

Conversation

@thatguywob

Copy link
Copy Markdown
Contributor

Summary

  • Return an explicit more flag from /api/matrix/lines, computed by looking for another allowed line after the page limit. Preserve the global seq value.
  • Use the flag to reveal caught-up reader paragraphs immediately when newer ring lines are filtered out, avoiding an empty follow-up request. Preserve pagination and the fallback for older servers without more.
  • Add endpoint and browser regression cases for filtered tails, full and short final pages, real pagination, and legacy responses; regenerate the dashboard asset and add a changelog fragment.

Fixes #16530

Testing

  • Not run locally: AGENTS.md requires CI validation for agent-authored changes.
  • gofmt -l on changed Go files: clean.
  • GOCACHE=/tmp/hive-16530-go-cache make dashboard-gen: succeeded.

β€” hive: backend=codex model=gpt-6.1-sol effort=high


🐝 Hive Agent: contributor | SHA: edf038b2d

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 4, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 4, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff against the v5 tree (matrix_source.go, api_matrix.go, both callers of SinceMatching, reader.js pull loop). The fix looks correct to me.

Server: SinceMatching (src/pkg/dashboard/matrix_source.go:423-441) now keeps scanning past a full page and reports true only when another kept line exists, so more is false when every newer ring entry is filtered out β€” exactly the case from #16530. The ring is capped at 2000 (matrix_source.go:25) and scanned under m.mu, so the extra tail scan is negligible. Since() is the only other caller and discards the flag; api_matrix.go:130 is the only consumer.

Client: reader.js:197 prefers data.more when it is a boolean and keeps the old returned < seq heuristic otherwise, so an older server still pages correctly; lastSeq = more ? returned : seq then jumps to the global high-water mark when nothing visible remains. The endpoint test table and the three new browser scenarios (trailing, page+trailing, legacy) cover the cases that matter. Generated static/index.html matches static-src/js/reader.js (one data.more occurrence each).

Sequencing note for maintainers (not a defect): #16513 (part 2/3 of the reader work) deletes this exact var more = … line and moves the reader to /api/matrix/lines?…&format=regions (served by handleCheckpointRegions), so the reader.js half of this PR is superseded once #16513 lands, and the two branches conflict in reader.js/matrix_source.go today (merge-tree shows markers). The server-side more flag stays useful for any other /api/matrix/lines caller. Suggest landing this first (small, green, fixes a shipped behaviour) and letting #16513 rebase.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

Read matrix_source.go, api_matrix.go and static-src/js/reader.js at edf038b plus the only other SinceMatching caller (Since). This looks correct to me.

  • SinceMatching (matrix_source.go:421-441) previously stopped at len(out) < limit, so a full page could not tell "more allowed lines remain" from "only filtered lines remain", and the client's returned < seq heuristic forced an extra pull that returned nothing. The new loop only returns more=true on encountering a kept line after the page is full, so filtered-out tails no longer trigger a catch-up; Since discards the new value and keeps its signature.
  • Client side (reader.js:196-198): more prefers data.more when it is a boolean and otherwise falls back to the old heuristic, so an older server still paginates. lastSeq = more ? returned : seq is unchanged, so when more=false the reader jumps to the global seq β€” which is exactly what the "only hidden lines" / "caught up" table cases in TestHandleMatrixLines_MoreCountsOnlyAllowedLines and the trailing browser scenario pin down.

Cost note (not a defect): the scan no longer exits early once the page is full; it walks the remaining ring until it finds one more kept line. For a ring of fixed size that is bounded and tiny next to the HTTP round-trip it removes.

stage 095-pre-merge-soak β†’ 060-reviewed: untrusted-author-role

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Server half looks correct; the client half edits code that #16513 (merged to v5 yesterday) removed, so this needs a rebase and a decision about what remains of #16530.

What I verified on 80ebdfb against v5:

  • src/pkg/dashboard/matrix_source.go:431-440 β€” the new loop keeps scanning after the page is full and returns more=true only on the next kept line, so a page full of allowed lines followed only by filtered ones correctly reports more=false. Cost is a full pass over the ring (matrixBacklogCap = 2000, matrix_source.go:22) under m.mu on every capped request instead of stopping at limit; fine at that size, just noting the trade-off.
  • api_matrix.go:81-85, 130-139 β€” More plumbed through; Since keeps its two-value shape so the other callers (matrix_source_test.go:62,313,317,321,473) are untouched.
  • api_matrix_test.go:421-456 covers the cases that matter (truncated / full final page / only hidden trailing lines / caught up). CI is green at this head apart from tide.

Why it cannot merge as-is (mergeable_state=dirty): git merge-tree origin/v5 80ebdfb conflicts in static-src/js/reader.js, static/index.html, reader_ui_test.go. After #16513 the v5 reader fetches /api/matrix/lines?since=…&format=regions (static-src/js/reader.js:193) and reads data.regions / data.latest with if (seq < Number(data.latest)) again = true (reader.js:211-212). The raw / returned / var more = … lines this PR changes no longer exist, and the lines-shaped responses with data.more are no longer consumed by the reader.

The regions path does not have the #16530 gap: checkpoint_source.go:289-315 applies no repo filter and advances Seq only through consumed regions (falling back to c.seq when none), so a caught-up client never sees seq < latest from hidden trailing entries.

Suggested next step for thatguywob: rebase onto v5, drop the three client-side hunks, and decide whether the More field + SinceMatching bool are still worth landing on their own for the plain lines format (other API consumers) β€” if yes, the Go half as written is fine; if the reader was the only consumer, #16530 may be closable as fixed-by-#16513.

hive/awaiting-ci: rebased onto v5 (now eca144d). #16513 moved the reader to format=regions and deleted the var more pull-loop line, so the reader.js / index.html / reader_ui_test.go half of this PR is superseded and was resolved to the v5 side; the server-side more flag on /api/matrix/lines (SinceMatching + endpoint test) still lands. Original commit and sign-off preserved; changelog fragment narrowed in a separate signed commit. CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16570 πŸ› fix(dashboard): render reader context lines in a neutral color same files src/pkg/dashboard/reader_ui_test.go
src/pkg/dashboard/static-src/js/reader.js
src/pkg/dashboard/static/index.html
#16514 ✨ feat(dashboard): replace Matrix presentation with Code activity same files src/pkg/dashboard/reader_ui_test.go
src/pkg/dashboard/static/index.html
#16500 ✨ feat(dashboard): replace Matrix presentation with Code activity same files src/pkg/dashboard/api_matrix.go
src/pkg/dashboard/matrix_source.go
src/pkg/dashboard/reader_ui_test.go
src/pkg/dashboard/static-src/js/reader.js
src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Ready for a human merge: verdict approved @edf038b, 67/67 checks green, DCO yes, soak 10m0s elapsed. Blocked only by role/untrusted (no trusted-author role).

The hive never approves on your behalf: /lgtm /approve (or merge) if you accept it.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the change against the tree at 80ebdfb, not just the diff. Looks correct to me.

What I verified

  • SinceMatching (src/pkg/dashboard/matrix_source.go:423-441) has exactly one non-test caller, api_matrix.go:130; Since keeps its old two-value signature, so the existing Since tests and matrix_persist_test.go:134 are unaffected. The loop now walks the whole ring (≀ matrixBacklogCap = 2000) only until it finds the limit+1-th kept line, so more is true iff another visible line remains β€” which is exactly what the client needs.
  • reader.js:195-201: lastSeq = more ? returned : seq is sound with the new flag. When more is false and the newest ring lines are filtered, lastSeq jumps to the global seq, so the next nudge pulls since=<global> and no empty catch-up request is issued. more === true implies len(out) == limit > 0, so returned is never 0 on the paging path. Legacy fallback (typeof data.more !== 'boolean') preserves the old returned < seq heuristic.
  • static/index.html:7683-7688 matches static-src/js/reader.js (regenerated asset).
  • go build ./pkg/dashboard/ and gofmt -l clean on the PR head; dashboard test shards 1/2/4 are already green in CI. (go test is blocked on agent pods β€” #9845.)
  • Intent: #16530 proposed "the server return … a more flag so the client need not guess"; this is that change, no extra scope. Fixes #16530 is the right link.

One suggestion (docs, low)

src/docs/api-reference/dashboard-misc.md:59 still says a paging client "should use the last returned line's seq as its next since, not the top-level seq". With this PR the documented contract becomes: continue from the last returned seq while more is true, otherwise resume from the top-level seq. Worth a one-line update in this PR so the API reference doesn't describe the pre-more behaviour.

Note for the maintainer: the conflicts-with-open-pr label is a sequencing question against the other open reader/dashboard PRs, not something wrong in this diff.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 4, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the branch conflicts with the base and needs a rebase

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob the hive has adopted this PR (stage/074-adopted-by-hive): CI stayed red for 2h after the nudge and the relay nudge went unanswered. The hive will rebase this branch on v5 and fix CI here; your commits and sign-off stay. Push a new head yourself at any time and the hive yields it back to you.

thatguywob and others added 2 commits November 4, 2026 05:59
Report whether another allowed line remains in a matrix page and use that signal to finish reader catch-up without an empty follow-up request. Keep the global sequence and legacy-server fallback.

Hive-Run: hivecommons#16530
Hive-Plan: issue-16530
Hive-Spec: hivecommons#16530
Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
The reader moved to format=regions on v5 (hivecommons#16513), so the client half of
this fix is superseded; the API change still lands for /api/matrix/lines.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Superseded by hive: re-authored as #16576 with the same commits and sign-off because its commits needed a sign-off only a hive-owned branch can carry. This PR is closed; if #16576 fails, hive reopens it.

@hivecommons-hive hivecommons-hive Bot added the hive/superseded Closed by Hive: a different merged PR already landed this change label Nov 4, 2026
@hivecommons-hive hivecommons-hive Bot closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. hive/superseded Closed by Hive: a different merged PR already landed this change size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[reviewer] Review backlog (style): Catch-up pull always costs one extra empty request when trailing lines are filtered

1 participant