feat: parallel index build with leader-merges architecture

[tjgreen42]

Summary

• Implements parallel index build where workers scan heap and build memtables, leader merges and writes segments
• Workers use double-buffered memtables in shared DSA memory to avoid blocking
• Leader performs all writes using dynamic relation extension (P_NEW)
• Serial and parallel builds now produce identical index output (same size, same format)

Architecture rationale

The key insight is moving all write responsibilities to the leader node:

1. Sequential writes: Leader writes segments sequentially, avoiding the heavy fragmentation that arose with earlier approaches where workers wrote their own segments
2. CPU-bound workload: Index building is mostly CPU-bound on the read path due to to_tsvector calls and memtable-population operations, so centralizing writes doesn't bottleneck throughput
3. No followup compaction: Previous approaches required a compaction step after workers finished; this design produces well-ordered segments directly

This architecture emerged from experimentation with several alternatives, informed by visualizations of page layout fragmentation.

Key changes

• Architecture: Workers fill memtables → signal ready → leader merges N buffers into single segment
• Threshold matching: Uses same tp_memtable_spill_threshold (32M postings) as serial build
• Compression: Applies delta+bitpacking compression to posting blocks (same as serial)
• Page boundary handling: Fixed dictionary entry writing across page boundaries
• Simplified page allocation: Uses P_NEW instead of pre-allocated pool (no GUC needed)
• Skip recovery docids: Removed unnecessary docid writes during CREATE INDEX (atomic operation doesn't need recovery). This applies to both serial and parallel builds, so indexes are now identical.

Benchmark (MS MARCO 8.8M docs)

Configuration	Time	Index Size
Serial	4:27	1190 MB
Parallel (4 workers)	1:26	1190 MB

3.1x faster with proper configuration. Index sizes now match exactly.

Note: Postgres requires maintenance_work_mem >= 64MB for parallel builds (the planner won't assign workers otherwise).

Testing

• make installcheck passes
• Tested on MS MARCO with 0, 1, 4 workers
• Index sizes match between serial and parallel builds

[SteveLauC]

Do we have a comparison between the old architecture (main branch) and this new one? From what I can see, the new architecture offers:

1. Less random I/O as leader is the only writer, and it writes sequentially (mostly)
2. Less small tail segments
3. Less chances to trigger compaction

[tjgreen42]

Do we have a comparison between the old architecture (main branch) and this new one? From what I can see, the new architecture offers:

1. Less random I/O as leader is the only writer, and it writes sequentially (mostly)
2. Less small tail segments
3. Less chances to trigger compaction

The main gist is to move write responsibilities to the leader node in order to make the writes mostly sequential (as you point out) and especially to avoid the heavy fragmentation that was arising with the previous approach, where workers wrote their own segments with a followup compaction step. Since we are mostly CPU bound during indexing (and mostly on the read path) due to the calls to to_tsvector and memtable-population operations, this turns out to give pretty good speedups in the # of workers. I had to experiment quite a bit to hit on this approach (several other PRs in flight that i'm now going to close) and add visualizations to see what was happening with the fragmentation. Ironically, I intended the viz for human consumption, but the robot started using them immediately and became the main consumer.

[SteveLauC]

Did a brief review of the PR, sorry for the bunch of comments 🫠

[tjgreen42]

Did a brief review of the PR, sorry for the bunch of comments 🫠

Thanks much @SteveLauC for going through the PR, it's helpful. I'd like to chat irl if you're game, ping me at tj@tigerdata.com.