Skip to content

[feat][store] Compaction-filter GC for the txn write CF with MVCC-awa… - #1482

Merged
yuhaijun999 merged 1 commit into
dingodb:mainfrom
visualYJD:yjd-dev
Aug 21, 2026
Merged

yuhaijun999 merged 1 commit into
dingodb:mainfrom
visualYJD:yjd-dev

Conversation

@visualYJD

Copy link
Copy Markdown
Contributor

…re auto compaction

  • Reject txn reads and prewrites older than the gc safe point (gc_enable_safe_point_read_check, ETXN_LT_GC_SAFE_POINT).
  • Add a TiKV-style txn GC compaction filter for the write CF: shadowed versions are dropped inside compaction without writing tombstones; orphan data CF keys are batch-deleted by a background cleaner; explicit periodic_compaction_seconds keeps cold bottom levels filtered.
  • Narrow filter-mode scan GC to delete-mark version groups, removed as raft-replicated per-user-key range deletes (delete_ranges_with_cf) so replicas whose compaction filters progressed differently still converge; defer the delete-mark flush until its version group ends.
  • Collect dingo.mvcc.* table properties on write CF SSTs and use them for MVCC-aware auto compaction: score regions by tombstones and reclaimable redundant versions, compact the dirtiest under a region cooldown and a per-round time budget.
  • Bound txn scan version-group crawls with one-shot seeks, both when crossing user keys (txn_scan_next_seek_bound) and when a snapshot read descends past versions newer than its start_ts.
  • Make the orphan cleaner gflags runtime adjustable via brpc /flags; log txn scan seek lock conflicts at INFO; enable the gc compaction filter and safe point read check by default (rollout requires every replica to understand raft delete_ranges_with_cf).
  • Bump dingo-store-proto to the rebased line carrying ETXN_LT_GC_SAFE_POINT and delete_ranges_with_cf.

…re auto compaction

- Reject txn reads and prewrites older than the gc safe point
  (gc_enable_safe_point_read_check, ETXN_LT_GC_SAFE_POINT).
- Add a TiKV-style txn GC compaction filter for the write CF: shadowed
  versions are dropped inside compaction without writing tombstones;
  orphan data CF keys are batch-deleted by a background cleaner;
  explicit periodic_compaction_seconds keeps cold bottom levels
  filtered.
- Narrow filter-mode scan GC to delete-mark version groups, removed as
  raft-replicated per-user-key range deletes (delete_ranges_with_cf) so
  replicas whose compaction filters progressed differently still
  converge; defer the delete-mark flush until its version group ends.
- Collect dingo.mvcc.* table properties on write CF SSTs and use them
  for MVCC-aware auto compaction: score regions by tombstones and
  reclaimable redundant versions, compact the dirtiest under a region
  cooldown and a per-round time budget.
- Bound txn scan version-group crawls with one-shot seeks, both when
  crossing user keys (txn_scan_next_seek_bound) and when a snapshot
  read descends past versions newer than its start_ts.
- Make the orphan cleaner gflags runtime adjustable via brpc /flags;
  log txn scan seek lock conflicts at INFO; enable the gc compaction
  filter and safe point read check by default (rollout requires every
  replica to understand raft delete_ranges_with_cf).
- Bump dingo-store-proto to the rebased line carrying
  ETXN_LT_GC_SAFE_POINT and delete_ranges_with_cf.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yuhaijun999
yuhaijun999 enabled auto-merge August 21, 2026 01:12

@yuhaijun999 yuhaijun999 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yuhaijun999
yuhaijun999 added this pull request to the merge queue Aug 21, 2026
Merged via the queue into dingodb:main with commit c5147a1 Aug 21, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants