Skip to content

geotiff: multiplier for HTTP range sizes for header metadata #500

@kylebarron

Description

@kylebarron

ideally for the header we would make "multiplicative growing" header requests. If we fetch 16kb and that's not enough, then fetch 64kb, then fetch 1MB, and so on. I'm not sure where that fix would be, if that's here in our geotiff package or upstream in @cogeotiff/core.


On multiplicative-growing reads

SourceChunk is stateless (@chunkd/middleware/build/src/middleware/chunk.js): it aligns each request to a fixed chunkSize. No memory of prior requests, no growth.

The natural place to add growing reads is @chunkd/middleware as a sibling middleware (e.g. SourceGrowingChunk({ initial, max, factor })):

  • Initial reads use a small chunk
  • Each subsequent fetch on the same source doubles the chunk up to max
  • Drops in alongside SourceChunk/SourceCache in the existing pipeline

Implementing this in @cogeotiff/core doesn't fit — it consumes a Source interface but doesn't define source middleware. The defaultReadSize it exposes (now wired through your prefetch option) is only used for the very first byte read, then it falls back to range fetches sized to whatever it needs.

Two paths:

  1. Upstream PR to @chunkd/middleware — best long-term, also benefits non-COG users.
  2. Implement locally as a custom SourceMiddleware in our geotiff package, then upstream when stable. The SourceMiddleware interface is just fetch(req, next), easy to implement.

See developmentseed/async-tiff#140

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions