feat: Search pipeline run API uses pagination cursor

[yuechao-qin]

TL;DR

Replaced offset-based pagination with cursor-based pagination for searching pipeline runs.

Old format (offset-based, base64-encoded)

The page_token was a base64-encoded JSON blob containing offset, filter, and filter_query:

base64({"offset": 10, "filter": null, "filter_query": null})
  = eyJvZmZzZXQiOiAxMCwgImZpbH=

API request for page 2:

GET /api/pipeline_runs?page_token=eyJvZmZzZXQiOiAxMCwgImZpbH=

New format (cursor-based, plain text)

The page_token is now a plain-text created_at~id cursor pointing to the last row of the previous page:

2024-02-01T09:00:00+00:00~018d8fff

API request for page 2:

GET /api/pipeline_runs?page_token=2024-02-01T09:00:00+00:00~018d8fff

What changed?

Functional

• Pagination mechanism: Switched from offset/limit to cursor-based pagination using created_at and id fields

Other

• Database optimization: Added composite index ix_pr_created_at_desc_id_desc on (created_at DESC, id DESC) for efficient cursor queries
• Query structure: Modified SQL queries to use tuple comparison (created_at, id) < (cursor_created_at, cursor_id) instead of OFFSET
• API compatibility: Maintained existing page_token parameter while changing internal implementation

How to test?

uv run pytest tests/test_api_server_sql.py tests/test_filter_query_sql.py

• Verify pagination works correctly with multiple pages of pipeline runs
• Test that cursor pagination maintains stable ordering even when new records are inserted between page requests
• Confirm no duplicate records appear across paginated results
• Validate that invalid page tokens raise appropriate errors
• Test pagination combined with filter queries

Why make this change?

Cursor-based pagination provides several advantages over offset-based pagination:

• Performance: Eliminates expensive OFFSET (reads all rows that are skipped) operations on large datasets
• Consistency: Prevents duplicate or missing records when data is inserted/deleted during pagination
• Scalability: Query performance remains constant regardless of page depth
• Stability: Results remain consistent even with concurrent modifications to the dataset

[yuechao-qin]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• feat: Add include_sql to Search Pipeline Run API #135
• feat: Search pipeline run API uses pagination cursor #131 👈 (View in Graphite)
• feat: Search within date range in pipeline run API #130
• feat: Search pipeline name in pipeline run API #129
• refactor: Convert legacy filter to be filter_query #128
• feat: Search created by user in pipeline run API #108
• feat: Search annotations in pipeline run API #107
• feat: Create Pydantic models for Search Pipeline Run API #106
• refactor: Search pipeline run refactoring and baseline unit tests #105
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Ark-kun]

One small advantage of opaque next_page_token is that the filter and cursor/offset are tied together.

[Ark-kun]

We can decode the created_at from the id. Do we need it in the cursor?

[morgan-wowk]

One small advantage of opaque next_page_token is that the filter and cursor/offset are tied together.

​Hmm...are we sure plain text is a good idea? I would rather keep it base64 encoded unless we explicitly want our users tampering with the search parameters. You are correct, filtering, page sizes, etc. impact the cursor. Sometimes, you use the encoded string to track additional metadata such as what page you're on. If the search conditions are tampered with, then metadata like current_page are no longer accurate.

I have hesitations about plain text and would be more comfortable if we kept it encoded for now.