Some highlights and reflections from achieving a 1m resolution viewshed-complete run of Galiano Island.
We always knew the main theme was going to be the sheer size of the raw viewshed output. Our first run took 36 hours and produced 19TB of data. There were some other issues with that attempt, so after implementing fixes and optimisations we got it down to 4 hours and 2.7TB. See https://galiano.alltheviews.world
It's worth noting that whereas for the world run our bottleneck is very much CPU, for a viewshed-based run like Galiano, the bottleneck is very much IO, therefore getting viewshed data onto disk fast enough. Sharded databases are essential here. We use 1 per thread, so for the GCP Turin machine we use we top out at 160 threads and so 160 databases. Thankfully Sqlite makes this easy. At first we were experimenting with merging all these databases together as a post-processing step. This made querying the data much easier. But in the end we found that keeping them separate actually has a number of benefits. Firstly it makes it possible to parallelise index creation across the 2.7TB of data. Secondly it avoids the need for an expensive merging step. And thirdly, although it leads to slightly more complicated querying logic, it likely leads to faster lookups due to asynchronous reads against many smaller chunks of the dataset. We currently have ~100ms API server responses.
There is still a question mark over the long-term economics of serving this data from a web server attached to terrabyte-sized SSDs. The web server itself needs very little CPU and RAM, but SSDs quickly raise the costs. It's not a problem for a few weeks or months. But we need to explore other ideas for the long-term. The main one being to manually pack and index the viewshed data into a custom but simple binary format and query it over a CDN bucket using HTTP Range requests.
The most immediate next step is the UI. We have some basic features for displaying multiple viewsheds of varying colours. But we need to see how this is actually used to know what would be a productive UI. It may be that even with the best UI there's just too much data to make it practical to find optimal viewsheds. There is still of course the whole other project of developing an algorithm that automatically searches for optimal viewsheds, but that's not the concern of this current iteration.
Some highlights and reflections from achieving a 1m resolution viewshed-complete run of Galiano Island.
We always knew the main theme was going to be the sheer size of the raw viewshed output. Our first run took 36 hours and produced 19TB of data. There were some other issues with that attempt, so after implementing fixes and optimisations we got it down to 4 hours and 2.7TB. See https://galiano.alltheviews.world
It's worth noting that whereas for the world run our bottleneck is very much CPU, for a viewshed-based run like Galiano, the bottleneck is very much IO, therefore getting viewshed data onto disk fast enough. Sharded databases are essential here. We use 1 per thread, so for the GCP Turin machine we use we top out at 160 threads and so 160 databases. Thankfully Sqlite makes this easy. At first we were experimenting with merging all these databases together as a post-processing step. This made querying the data much easier. But in the end we found that keeping them separate actually has a number of benefits. Firstly it makes it possible to parallelise index creation across the 2.7TB of data. Secondly it avoids the need for an expensive merging step. And thirdly, although it leads to slightly more complicated querying logic, it likely leads to faster lookups due to asynchronous reads against many smaller chunks of the dataset. We currently have ~100ms API server responses.
There is still a question mark over the long-term economics of serving this data from a web server attached to terrabyte-sized SSDs. The web server itself needs very little CPU and RAM, but SSDs quickly raise the costs. It's not a problem for a few weeks or months. But we need to explore other ideas for the long-term. The main one being to manually pack and index the viewshed data into a custom but simple binary format and query it over a CDN bucket using HTTP Range requests.
The most immediate next step is the UI. We have some basic features for displaying multiple viewsheds of varying colours. But we need to see how this is actually used to know what would be a productive UI. It may be that even with the best UI there's just too much data to make it practical to find optimal viewsheds. There is still of course the whole other project of developing an algorithm that automatically searches for optimal viewsheds, but that's not the concern of this current iteration.