Inserting large datasets row-by-row (or even in small batches) creates a severe bottleneck once you cross ~10K rows. The overhead of individual round-trips, transaction logging, and index maintenance per statement adds up fast.
The hypothesis here: PostgreSQL COPY outperforms batched INSERT by an order of magnitude at scale. This module should prove (or disprove) that with real numbers.
Reproduce the bottleneck first — a naive loop inserting rows one-at-a-time — then implement bulk strategies: JDBC batch inserts, COPY via PgConnection, and possibly Spring Data's saveAll with batch sizing. Benchmark each at 1K, 10K, 100K rows. Correctness check: row counts and data integrity post-insert.
Must satisfy AC-1 through AC-5. Standard module layout with benchmark artifacts under docs/.
Inserting large datasets row-by-row (or even in small batches) creates a severe bottleneck once you cross ~10K rows. The overhead of individual round-trips, transaction logging, and index maintenance per statement adds up fast.
The hypothesis here: PostgreSQL
COPYoutperforms batchedINSERTby an order of magnitude at scale. This module should prove (or disprove) that with real numbers.Reproduce the bottleneck first — a naive loop inserting rows one-at-a-time — then implement bulk strategies: JDBC batch inserts,
COPYvia PgConnection, and possibly Spring Data'ssaveAllwith batch sizing. Benchmark each at 1K, 10K, 100K rows. Correctness check: row counts and data integrity post-insert.Must satisfy AC-1 through AC-5. Standard module layout with benchmark artifacts under
docs/.