Added shard prefetching to speed up loading from network filesystems by jpc · Pull Request #16 · HumeAI/wsds

jpc · 2026-01-26T18:09:45Z

📚 PR Stack

#	Branch	Base	PR
1	`jpc/preloading`	`main`	#16 ◀️
2	`jpc/validate-shards-in-sql`	`jpc/preloading`	#18
3	`jpc/shard-pipe`	`jpc/validate-shards-in-sql`	#19
4	`jpc/sql-without-index`	`jpc/shard-pipe`	#20
5	`jpc/special-shard-columns`	`jpc/sql-without-index`	#21
6	`jpc/remove-in-progress`	`jpc/special-shard-columns`	#22
7	`jpc/fix-jupyter-repr`	`jpc/remove-in-progress`	#23
8	`jpc/sql-select-dotted`	`jpc/fix-jupyter-repr`	#24
9	`jpc/optimize-durations`	`jpc/sql-select-dotted`	#25
10	`jpc/source-links`	`jpc/optimize-durations`	#26
11	`jpc/audio-qol`	`jpc/source-links`	#27
12	`jpc/warn-subsampling-in-dataloader`	`jpc/audio-qol`	#28
13	`jpc/wsds-inspect-head`	`jpc/warn-subsampling-in-dataloader`	#29
14	`jpc/feather-index`	`jpc/wsds-inspect-head`	#30
15	`jpc/indexer`	`jpc/feather-index`	#31
16	`jpc/keys-diff`	`jpc/indexer`	#32
17	`jpc/fields-cleanup`	`jpc/keys-diff`	#33
18	`jpc/s3-shards`	`jpc/fields-cleanup`	#34
19	`jpc/indexing-fixes`	`jpc/s3-shards`	#35
20	`jpc/validate-keys-on-the-fly`	`jpc/indexing-fixes`	#36
21	`jpc/repr-missing-last`	`jpc/validate-keys-on-the-fly`	#37
22	`jpc/drop-slots`	`jpc/repr-missing-last`	#38

Added shard prefetching to speed up loading from network filesystems

06e0707

shahbaz-humeai approved these changes Mar 11, 2026

View reviewed changes

shahbaz-humeai merged commit d8b25c8 into main Mar 13, 2026