Skip to content

feat(api): add cursor pagination to external API v2 search - #3247

Open
vinzee wants to merge 3 commits into
ClickHouse:mainfrom
vinzee:va/cursor-based-pagination
Open

vinzee wants to merge 3 commits into
ClickHouse:mainfrom
vinzee:va/cursor-based-pagination

Conversation

@vinzee

@vinzee vinzee commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Problem

/api/v2/search pages with LIMIT/OFFSET, so each page costs more than the last and offset stops at 10,000.

Solution

Adds an optional cursor request field and a nextCursor response field. offset callers are unchanged.

Cursor paging uses about 30% less ClickHouse CPU than offset paging on real pipedream-logs queries, with identical results. For queries sorted by timestamp, OFFSET stays small however many pages you fetch, and the 10,000-row cap no longer applies.

Paging through one hour of logs, 1,000 rows per page:

// 1. First page: same body as an offset request, plus "cursor": "start"
POST /api/v2/search
{ "sourceId": "65f...", "startTime": "2026-10-25T10:00:00Z", "endTime": "2026-10-25T11:00:00Z",
  "where": "ServiceName:checkout", "maxResults": 1000, "cursor": "start" }
-> { "data": [ /* 1000 rows, newest first */ ], "rows": 1000, "nextCursor": "eyJ2Ijox...Q3a9" }

// 2. Next page: same body, cursor replaced with the returned nextCursor
{ ..., "cursor": "eyJ2Ijox...Q3a9" }
-> { "data": [ /* 212 rows */ ], "rows": 212, "nextCursor": "eyJ2Ijox...b7Kc" }  // short page, not the end

// 3. Keep going until nextCursor is null
{ ..., "cursor": "eyJ2Ijox...b7Kc" }
-> { "data": [ /* 640 rows */ ], "rows": 640, "nextCursor": null }               // done
  • Send "start", then pass back each nextCursor until it is null. An empty or short page does not mean the end.
  • When cursor is set, offset is ignored. The cursor is opaque.
  • Rows sharing a timestamp can be skipped or repeated unless orderBy breaks the tie, as with offset.
  • Errors are 400s: a malformed or tampered cursor, or a cursor reused with a different query or range.

How it works

  • The range is read in time windows (15m, 6h, 6h, 12h, then 24h). Each page reads one window; boundaries are whole seconds, so second-precision columns like TimestampTime don't lose rows.

  • Within a window, each full page shrinks the window to the last row's second (its end for DESC, its start for ASC), so OFFSET only skips that second's rows and page cost stays flat in dense ranges.

  • The cursor is base64url(payload).base64url(HMAC), with payload:

    Field Meaning
    v Cursor format version
    k Window number: which time window the next page reads from
    ws, we Current window's bounds; the window shrinks after each full page
    o Rows to skip: those already returned from the last row's second
    f Fingerprint of sourceId, where, whereLanguage, select, orderBy; a mismatch is a 400
    s, e The query's time range, fixed on the first page so a defaulted endTime doesn't move
  • Each page computes the next window from k, ws/we and s/e alone: its near edge is the current window's far edge, and its far edge is the range edge offset by the total length of the first k + 1 windows. Nothing is built up front, so the range has no cap.

  • Cursors are HMAC-signed with EXPRESS_SESSION_SECRET, so they can't be edited; replicas need a shared value.

  • Queries sorted first by an exact timestamp, or by toStartOfSecond/Minute/FiveMinutes/FifteenMinutes(...), use windows aligned to that key. Others (hour/day keys, non-time keys, DISTINCT selects) read the whole range with plain offsets and keep the 10,000 cap.

Testing

  • Unit and integration tests (search.int): every row returned exactly once across window boundaries, dense and burst seconds, compound timestamp expressions, both directions, tampered cursors.
  • Against dev ClickHouse, compared with pipedream-logs' offset paging: identical rows (up to 130k), and 29–41% less ClickHouse CPU.

Screenshots or video

n/a

How to test on Vercel preview

n/a

References

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants