Repository navigation
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem
/api/v2/searchpages withLIMIT/OFFSET, so each page costs more than the last andoffsetstops at 10,000.Solution
Adds an optional
cursorrequest field and anextCursorresponse field.offsetcallers are unchanged.Cursor paging uses about 30% less ClickHouse CPU than offset paging on real pipedream-logs queries, with identical results. For queries sorted by timestamp,
OFFSETstays small however many pages you fetch, and the 10,000-row cap no longer applies.Paging through one hour of logs, 1,000 rows per page:
"start", then pass back eachnextCursoruntil it isnull. An empty or short page does not mean the end.cursoris set,offsetis ignored. The cursor is opaque.orderBybreaks the tie, as withoffset.How it works
The range is read in time windows (15m, 6h, 6h, 12h, then 24h). Each page reads one window; boundaries are whole seconds, so second-precision columns like
TimestampTimedon't lose rows.Within a window, each full page shrinks the window to the last row's second (its end for DESC, its start for ASC), so
OFFSETonly skips that second's rows and page cost stays flat in dense ranges.The cursor is
base64url(payload).base64url(HMAC), with payload:vkws,weofsourceId,where,whereLanguage,select,orderBy; a mismatch is a 400s,eendTimedoesn't moveEach page computes the next window from
k,ws/weands/ealone: its near edge is the current window's far edge, and its far edge is the range edge offset by the total length of the firstk + 1windows. Nothing is built up front, so the range has no cap.Cursors are HMAC-signed with
EXPRESS_SESSION_SECRET, so they can't be edited; replicas need a shared value.Queries sorted first by an exact timestamp, or by
toStartOfSecond/Minute/FiveMinutes/FifteenMinutes(...), use windows aligned to that key. Others (hour/day keys, non-time keys,DISTINCTselects) read the whole range with plain offsets and keep the 10,000 cap.Testing
search.int): every row returned exactly once across window boundaries, dense and burst seconds, compound timestamp expressions, both directions, tampered cursors.Screenshots or video
n/a
How to test on Vercel preview
n/a
References