test/ci: Add optimize endpoint + trim image build to amd64-only with PR SHA push#7869
Draft
test/ci: Add optimize endpoint + trim image build to amd64-only with PR SHA push#7869
Conversation
Adds POST /tests/<dataset>/optimize which calls OPTIMIZE TABLE … FINAL
on every ClickHouse table belonging to that dataset.
ClickHouse's ReplacingMergeTree deduplicates rows (tombstones from
deletions, replacement rows from group merge/unmerge) in the background.
Under high parallel load, background merges lag significantly, causing
test assertions to see stale data.
OPTIMIZE TABLE FINAL forces immediate synchronous deduplication.
Usage from Sentry tests:
requests.post(f"{settings.SENTRY_SNUBA}/tests/events/optimize")
requests.post(f"{settings.SENTRY_SNUBA}/tests/groupedmessage/optimize")
Changes:
- Remove arm64 from all build jobs (amd64-only); CI runners are all amd64
and building arm64 doubles build time with no benefit for CI
- Make `assemble` run on PRs (not just master/release) so that
ghcr.io/getsentry/snuba:{sha} is available for Sentry's devservices
to reference in integration tests and local development overrides
- `nightly` tag is still only applied on master pushes
- Collapse multi-matrix jobs into single jobs (no matrix needed for one arch)
- `self-hosted-end-to-end` now depends on `assemble` so the SHA image
is guaranteed to exist before the e2e run
joshuarli
changed the title
joshuarli
force-pushed
the
test-optimize-endpoint
branch
from
2e86853 to
70577b4
Compare
joshuarli
added a commit
to getsentry/sentry
that referenced
this pull request
Apr 11, 2026
Points all snuba devservices dependencies at the test-optimize-endpoint PR branch (getsentry/snuba#7869) which: 1. Adds POST /tests/{dataset}/optimize for synchronous ClickHouse dedup 2. Pins the image to 48d65526d...-amd64 which includes the endpoint Reverts to master once the PR is merged and nightly is cut.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related changes to make Snuba PRs testable against Sentry's shuffle-tests workflow.
1.
POST /tests/{dataset}/optimizeendpointAdds a new test-mode endpoint that calls
OPTIMIZE TABLE … FINALon every ClickHouse table in a dataset.ClickHouse's
ReplacingMergeTreededuplicates rows (tombstones from deletions, replacement rows from group merge/unmerge) in the background. Under high parallel load (e.g. 16-shard test runs), background merges lag significantly, causing test assertions to see stale data.OPTIMIZE TABLE FINALforces immediate synchronous deduplication, making ClickHouse state consistent for subsequent queries.Usage from Sentry tests:
2. Image workflow: amd64-only + SHA tag on PRs
ghcr.io/getsentry/snuba:{sha}on PRs —assemblenow runs onpull_requestevents, so the SHA-tagged image is available for Sentry's devservices to referencenightlytag still only applied on master pushesself-hosted-end-to-endnow depends onassembleWhy the image change is needed: PRs previously built
{sha}-amd64+{sha}-arm64but no merged manifest — theassemblejob was gated to master only. Sentry's devservices config referencesghcr.io/getsentry/snuba:{sha}which didn't exist for PR builds.Test plan
ghcr.io/getsentry/snuba:{sha}on this PRnightlyis NOT re-tagged on a PR buildnightlyIS re-tagged after merge to master