ideally for the header we would make "multiplicative growing" header requests. If we fetch 16kb and that's not enough, then fetch 64kb, then fetch 1MB, and so on. I'm not sure where that fix would be, if that's here in our geotiff package or upstream in @cogeotiff/core.
On multiplicative-growing reads
SourceChunk is stateless (@chunkd/middleware/build/src/middleware/chunk.js): it aligns each request to a fixed chunkSize. No memory of prior requests, no growth.
The natural place to add growing reads is @chunkd/middleware as a sibling middleware (e.g. SourceGrowingChunk({ initial, max, factor })):
- Initial reads use a small chunk
- Each subsequent fetch on the same source doubles the chunk up to max
- Drops in alongside SourceChunk/SourceCache in the existing pipeline
Implementing this in @cogeotiff/core doesn't fit — it consumes a Source interface but doesn't define source middleware. The defaultReadSize it exposes (now wired through your prefetch option) is only used for the very first byte read, then it falls back to range fetches sized to whatever it needs.
Two paths:
- Upstream PR to @chunkd/middleware — best long-term, also benefits non-COG users.
- Implement locally as a custom SourceMiddleware in our geotiff package, then upstream when stable. The SourceMiddleware interface is just fetch(req, next), easy to implement.
See developmentseed/async-tiff#140
ideally for the header we would make "multiplicative growing" header requests. If we fetch 16kb and that's not enough, then fetch 64kb, then fetch 1MB, and so on. I'm not sure where that fix would be, if that's here in our geotiff package or upstream in
@cogeotiff/core.On multiplicative-growing reads
SourceChunk is stateless (@chunkd/middleware/build/src/middleware/chunk.js): it aligns each request to a fixed chunkSize. No memory of prior requests, no growth.
The natural place to add growing reads is
@chunkd/middlewareas a sibling middleware (e.g.SourceGrowingChunk({ initial, max, factor })):Implementing this in @cogeotiff/core doesn't fit — it consumes a Source interface but doesn't define source middleware. The defaultReadSize it exposes (now wired through your prefetch option) is only used for the very first byte read, then it falls back to range fetches sized to whatever it needs.
Two paths:
See developmentseed/async-tiff#140