Add color table support to raster IO - #40
Merged
Merged
Conversation
Raster.save and write_raster accept a color_table keyword that attaches a discrete color palette to the written raster. It takes a dict mapping values to (r, g, b) or (r, g, b, a) components, an array-like of shape (N, 3) or (N, 4) where row i is the color for value i, or the path to an existing raster whose first-band table is copied. No color table state is tracked on the Raster; it is supplied at write time only. GDAL names this structure a color table (GetColorTable/SetColorTable, gdalinfo's "Color Table (RGB with 256 entries)"), and "palette" names the interpretation that a band's values index one. The TIFF tag and rasterio's API both say colormap, but that word means a continuous ramp to most of Python geo, so color_table follows the existing house style of preferring the clearer term over the wrapped library's, as null_value already does for nodata. The table is written in an update pass after the data, which is also what sets the file's palette photometric interpretation. Requesting that interpretation as a creation option on top of this only reserves a second color table block that nothing reads, so the tag is left to the update pass and pinned by the tests instead. The COG driver has no palette creation option, so that path applies the table to its staging file, which the copy carries over. GDAL fails silently in the cases where a color table cannot be stored, so they are rejected up front: tables only work on uint8 and uint16 bands, and one written to a multi-band file is dropped without an error. Entries outside the target dtype's index range are dropped only when the table came from a file, since GDAL pads a table it hands back out to the source dtype's full range; that padding is an artifact rather than intent, so copying a uint16 source table onto a uint8 raster works. Alpha is kept for the palette formats that store it and forced opaque, with a warning naming the driver, for those that do not. Averaging or interpolating overview resampling blends the indices into ones that no longer correspond to the original colors, so that combination warns when overviews will actually be built.
save() could copy a color table from a file by path, but there was no way to see one. rioxarray discards color tables on read, so inspecting or editing a palette meant dropping to rasterio. read_color_table(path, band=1) returns the table as a dict that can be handed straight back to save(), so an existing palette can be inspected, edited entry by entry, or used as the base for a new one. It takes a path rather than a Raster because a Raster deliberately does not carry a table, and its recorded source goes stale after any operation. The table is returned exactly as stored, which for a GeoTIFF means padded out to the band dtype's full index range: an eight class table reads back with 256 entries. The padding is not filtered because it cannot be done safely. An undefined entry is opaque black, which is indistinguishable from a deliberate black entry, and a black class is common in categorical data where it marks background. Trimming would need the values a raster actually uses, which means reading the data, so it is left to the caller. An out of range band raises rasterio's IndexError rather than being wrapped, since "No such band index: 2" already says what is wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
color_tablekeyword toRaster.saveandio.write_rasterthat attaches adiscrete color palette to the written raster.
Accepted specs: a
dictmapping cell values to(r, g, b)or(r, g, b, a)components in 0-255, an array-like of shape
(N, 3)or(N, 4)where rowiisthe color for value
i, or a path to an existing raster whose first-band table iscopied.
save_chunksforwards it, so each tile gets the same table.And a
read_color_tablefor the other direction, exported at the top level:rioxarray discards color tables on read, so before this there was no way to see a
palette through raster_tools at all -- you could copy one by path but not inspect or
edit it. The result feeds straight back into
save. It takes a path rather than aRasterbecause aRasterdeliberately does not carry a table, and its recordedsource goes stale after any operation.
Additive only: with
color_table=Nonethe write path is unchanged, and a test pinsthat the output has no palette and stays
min-is-black.Why this scope
No color table state is tracked on the
Raster. The palette is supplied at writetime and nothing else in the library knows about it.
This means
open() -> save()does not preserve a palette, which is the obviousquestion to ask of this PR. It is deliberate, not an oversight. rioxarray discards
color tables on read (its only mention of them is a no-op stub in
merge.py), socarrying one would mean reading it out separately and deciding how it survives
every operation in the library. Most operations must drop it -- arithmetic, focal
means, and interpolating reprojection all produce values that no longer index the
table -- while clipping, band selection, and nearest/mode resampling should keep it.
That is a much larger change with real semantics to get wrong, and it is not needed
for the case this addresses: writing an already-classified raster out with colors
attached.
If it is picked up later,
_ds.attrsis the carrier to use, becausemake_raster_dsbuilds a fresh Dataset per operation and so drops attrs by default -- the safe
behavior for value-metadata, meaning code is only needed where preservation is
wanted rather than where it is dangerous.
Naming
GDAL calls this structure a color table (
GetColorTable/SetColorTable,gdal.ColorTable, andColor Table (RGB with 256 entries)ingdalinfooutput)."Palette" names the interpretation -- that a band's values index one.
The TIFF tag and rasterio's API both say "colormap", but in Python geo that word
overwhelmingly means a matplotlib continuous ramp, and the collision is not
theoretical:
save(colormap="viridis")would have been read as a file path andfailed with
FileNotFoundError: Path does not exist: 'viridis'.color_tablefollows the existing house style of preferring the clearer term over the wrapped
library's, as
null_valuealready does fornodata.Implementation notes
GDAL fails silently in most cases where a color table cannot be stored, so those are
rejected up front rather than letting the palette disappear:
NULL color tableinside GDAL.ValueError, leavingcolorinterpas(palette, green, blue).files without a palette all raise
ValueError/RasterIOError.Three behaviors that are less obvious:
range, so an eight class table reads back with 256 entries (65536 for uint16).
On the write side those extras are an artifact rather than intent, so entries
outside the target dtype's range are dropped only when the table was read from
a file -- copying a uint16 source table onto a uint8 raster works instead of
failing on index 256.
read_color_tablereturns them, because the padding cannotbe filtered safely: an undefined entry is opaque black, which is byte-identical to
a deliberate black entry, and a black class is common in categorical data where it
marks background. Trimming needs the values a raster actually uses, which means
reading the data, so it is left to the caller.
with a warning naming the driver. PNG and GIF palettes do store alpha, so it is
kept there.
.pngwas already a reachable output path (it is not inWRITE_NOT_IMPLEMENTED_EXTS); this stops that path silently destroyingtransparency the format can hold, rather than adding PNG support.
indices that no longer correspond to the original colors, so that combination
warns. The check is case-insensitive because the COG translator lowercases the
method before GDAL sees it, and it only fires when overviews will actually be
built, so a raster too small for the auto chain does not warn about output that
will not exist.
The table is written in an update pass after the data, which is also what sets the
file's palette photometric interpretation. Requesting that interpretation as a
creation option on top of it only reserves a second color table block that nothing
reads -- measured at exactly 1548 wasted bytes per GTiff, with COG output
byte-for-byte identical -- so the tag is left to the update pass and pinned by the
tests instead. The COG driver has no palette creation option, so that path applies
the table to its staging file, which
rasterio.shutil.copycarries over.Testing
51 new cases in the "Color tables" and "read_color_table" sections of
tests/test_write.py, covering each spec form, uint8/uint16, COG, PNG,save_chunksforwarding, read/edit/write round-trips, cell values beingundisturbed, and every rejection path. Full suite: 2527 passed.
Conformance is asserted by reading TIFF tag 262 directly (
_tiff_photometric),because rasterio exposes neither
dataset.photometricnorprofile["photometric"]for these files -- both return
Noneeven for a correctly tagged palette -- andcolorinterpreportspaletteeven when the tag says min-is-black.Every guard was mutation-checked: 12 of 12 seeded mutations (band-count source,
padding skip, alpha gating, overview predicate, case sensitivity, error type, dtype
coercion, dtype restriction, multi-band guard, and both table-write call sites) are
caught by a named test.
Public surface
read_color_tableis added toraster_tools.__all__and gets a "Color Tables"section in
docs/reference/top_level.rst.Raster.savegains thecolor_tablekeyword. It is already in the docsautosummary, so the parameter documentation flows through from the docstring.
normalize_color_table(color_table, dtype, driver=None)andCOLOR_TABLE_DTYPESare non-underscore names inraster_tools.io, alongsidethe existing
write_rasterandnormalize_null_value, but are not exported atthe top level. Happy to make them private if that is preferred.
Not included
Attaching a color table to a
Rasterso it survives operations, categorical/RATsupport, and continuous ramps -- which a GeoTIFF cannot represent at all, and would
want a separate helper expanding a raster plus a ramp into RGB bands.