Stop losing data when split-and-merge compaction stalls mid-way

[1linkovdim]

Problem

Split-and-merge compaction can leave a group's sharded split output at level 2 permanently. The merge stage only runs when a shard holds two or more blocks, so a range whose split produced exactly one block per shard is never promoted.

That is not a rare corner. It happens whenever ingestion is scheduled rather than continuous: the "don't compact the most recent blocks prematurely" guard (rangeEnd <= highestMaxTime) rejects the merge job for the last open range, and once ingestion stops for the day no later block arrives to un-gate it.

Once the level 1 ancestors are deleted, those level 2 blocks are the only copy of the data — and the querier dropped them unconditionally. pruneSupersededBlocks treated any sharded block below the deduplication level as superseded, with no check that anything survived to serve the range. An hour of profiles silently returned nothing, with no error and no log line.

Two details made it worse:

• The second clause of the guard, job.maxTime()-job.minTime() == job.rangeLength(), is inherited from Mimir where a TSDB block's MaxTime is the range boundary. In Pyroscope MaxTime is the newest profile's timestamp, so the clause is effectively unsatisfiable and the guard collapses to rangeEnd <= highestMaxTime.
• highestMaxTime was computed over the whole tenant while planning is per main group, so an unrelated group's fresher data decided whether a group whose range was still open got compacted.

Changes

Querier — serve intermediate blocks when there is no fallback (pkg/querier/replication.go)

pruneSupersededBlocks now prefers a block's ancestors over the intermediate block only when those ancestors are still present. If they are gone, the intermediate block is kept and the query deduplicates, with a warning so the stalled compaction is visible.

The shard-completeness check is split in two: once at the deduplication level before the collapse (so an incomplete deduplicated set is pruned while its ancestors are still around as a fallback), and once at level 0 afterwards over whatever survived. The second pass is what preserves the existing protection against serving a partial set — if only some shards have a usable fallback, the plan is empty rather than half the data.

A single definition of the deduplication level (pkg/phlaredb/block/dedup.go)

Level 2, or 3 when split sharding is enabled, was hardcoded on both the querier and compactor sides. It is now one shared definition, so the two cannot disagree about which blocks are authoritative.

Compactor — promote a lone sharded block (pkg/compactor/split_merge_grouper.go, pkg/phlaredb/compact.go)

A merge job is now planned for a single sharded block that is still below the deduplication level, instead of skipping it forever. This terminates because the output is one level higher. The len(shardBlocks) < 2 check stays for every other case — it is the planner's only termination condition.

phlaredb.CompactWithSplitting rejects single-block compactions as no-ops; that guard is relaxed only via an explicit AllowSingleBlock option, which the block compactor sets solely for this deliberate promotion.

Compactor — scope the premature-compaction guard to the group (pkg/compactor/split_merge_grouper.go)

highestMaxTime is taken from the most recent block of the group being planned rather than of the whole tenant. Groups are independent streams written by unrelated services on unrelated schedules. This makes the filter strictly stricter, never more permissive.

Tests

• pkg/querier/replication_test.go — intermediate sharded blocks kept when ancestors are gone (with deduplication) and dropped when they are not; dropped when a deduplicated block supersedes them; the partial-fallback case still yields an empty plan; the warning is emitted only when an intermediate block actually has to be served.
• pkg/compactor/split_merge_grouper_test.go — a lone sharded block below the deduplication level is merged; one at or above it, or with no compaction metadata, is not; a group's open range is not compacted just because another group has fresher blocks, and is compacted once its own range closes.
• pkg/phlaredb/compact_test.go — single-block compaction is rejected by default, allowed with AllowSingleBlock and produces a block one level higher, and an empty input is always rejected.

go test ./pkg/querier/... ./pkg/compactor/... ./pkg/phlaredb/... and make lint pass.

🤖 Generated with Claude Code