Skip to content

FIX import-tools docs to describe the normalisation the code performs - #70

Merged
mhdned merged 6 commits into
mainfrom
hotfix/nodata-safe-calculators
Oct 17, 2026
Merged

FIX import-tools docs to describe the normalisation the code performs#70
mhdned merged 6 commits into
mainfrom
hotfix/nodata-safe-calculators

Conversation

@mhdned

@mhdned mhdned commented Oct 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

The Geoeye section said the tool reads the sensor bit depth and divides
by 2^b - 1 via img_as_float. It does neither: it applies one min-max
stretch over the whole stack, so a single band spans only part of
[0, 1] and its scale depends on the other bands in the file.

The Landsat8 section said the raw mode returns uint16; it returns
float64 with nodata as NaN. The "rgb" and "infrared" modes stretch each
band on its own, which discards the brightness relationships between
bands. Any other exportType falls back to the raw stack.

mhdned and others added 6 commits September 21, 2026 06:06
@mhdned
FIX process() path validation, stacked band input, and nodata fill fo…
a118b47 
…r issue #66
Mohammad Kiani
ADD shared nodata masking helpers for the calculators
591adec 
Masking a declared nodata fill to NaN on read fixes the cases where the fill
was silently treated as data, but many of the algorithms the calculators wrap
cannot take NaN at all: OpenCV's median filter rejects float64, scikit-image
cannot build a histogram of NaN, and scikit-learn's KMeans and SVC refuse
missing values. On its own the masking trades a wrong answer for a crash on
the most ordinary input there is, a scene with a fill margin.

Add fezrs/utils/nodata_handler.py so every calculator applies one rule rather
than an ad hoc patch each:

  * inputs without declared nodata are untouched - no mask is built and the
    output is exactly what it was;
  * when inputs declare nodata the output is float with NaN over it, the
    computation having run on valid pixels only;
  * windowed operations also report NaN wherever the window touches nodata,
    since a filtered or texture value computed partly from fill is not a
    measurement, and at a scene edge it manufactures a false boundary.

invalid_mask() returns None when every pixel is valid, which is the signal to
leave a computation alone. fill_invalid() gives NaN-intolerant algorithms a
finite stand-in - the valid median, which unlike 0 or the minimum does not drag
a histogram or local statistic to an extreme - that is masked back out
afterwards by apply_nodata().
Mohammad Kiani
FIX to_raster() writing channel-last output with the wrong axis order
3469bdf 
to_raster() treated every 3-D array as (bands, height, width). The RGB and
colour-space tools - OriginalRGB, EqualizeRGB, AdaptiveRGB, GammaRGB, HSV,
IRHSV and Landsat8 - return (height, width, channels), so a 998x998 RGB
composite was written as 998 bands of 998x3 pixels at the source transform.
The file opened without complaint and was wrong everywhere. A reviewer hit the
same thing from the other side: a stacked input produced a (10, 14, 2) array
that was written as 10 bands, height 14, width 2.

The source band already fixes the grid the output has to sit on, which settles
the axis order without guessing: band-first output is written as is,
channel-last output is moved to band-first, and anything matching the grid on
neither axis order is refused rather than written misregistered at the source
transform. A 2-D output must match the grid exactly for the same reason.

The smoke test for to_raster() missed this because it only asserted that a CRS
and some band existed. It now asserts the written height and width equal the
input grid and that the band count equals the computed channels; reverting
this fix fails it for all seven RGB tools. One existing test wrote a 2x2 label
map against a stubbed 4x4 grid and now uses a 4x4 map.
Mohammad Kiani
FIX calculators that crash or misreport on nodata-declared input
034f278 
Masking a declared nodata fill to NaN on read (a118b47) fixed the silent cases
for most calculators, but many of the algorithms they wrap cannot take NaN, so
on a scene with a fill margin five calculators went from a wrong answer to a
crash, and several others still reported the fill as data. Run across all 37
calculators on inputs with a declared -9999 fill:

                               main    a118b47   this change
  misbehaving calculators     34/37     14/37        0/37

Newly crashing, now fixed:
  * MedianCalculator - OpenCV's median filter rejects float64.
  * EqualizeCalculator, EqualizeRGBCalculator - equalize_hist cannot histogram
    NaN; it takes a mask argument that was being passed as None.
  * KMeansCalculator, SVMCalculator - scikit-learn refuses missing values.
    Both now fit and predict on valid pixels only; left in, the fill would take
    a cluster of its own or be given a land-cover class. An SVM training sample
    placed on nodata is rejected with a clear error.

Still reporting the fill as data, now NaN:
  * SobelCalculator, AdaptiveCalculator, AdaptiveRGBCalculator, HSVCalculator,
    IRHSVCalculator.
  * BurnCalculator - a boolean map cannot hold "no data", so the fill compared
    as False and was reported as unburned, a confident claim about ground
    nobody observed. With nodata declared it is now float: 1, 0 or NaN.
  * MeanCalculator - cv2.blur is a running-sum box filter, so once a NaN
    entered the sum everything to the right of the fill in each row came back
    NaN, destroying valid data along with the fill.
  * GLCMCalculator - the fill no longer sets one end of the quantization
    range: the valid pixels of a scene with a fill margin now quantize
    identically to the same pixels without one. Windows that touch the fill
    are NaN, since along a scene edge their contrast reads exactly like a
    lithological contact.

Every change follows the rule in nodata_handler: inputs without declared
nodata produce exactly the output they did before.

Two pre-existing defects in the same functions, independent of nodata:

  * SobelCalculator passed an output depth of 0, which OpenCV reads as CV_8U
    rather than "same as input". On 16-bit data every negative gradient was
    clipped to 0 and the rest saturated at 255 - 44% and 41% of pixels on a
    synthetic 16-bit scene against a true range of -9874..9921 - erasing the
    sign that lineament mapping depends on. The gradient is now float64.
  * MedianCalculator raised cv2.error for any kernel above 5 on 16-bit data,
    since OpenCV supports larger kernels on uint8 only. A dtype-safe median
    keeps the OpenCV path, bit-identical, where it is supported and uses
    scikit-image's exact median otherwise.

Also make GeoeyeCalculator.process() return its result, as every other
process() does.

tests/nodata_test.py runs every calculator's execute(), histogram_export()
and to_raster() on nodata-declared GeoTIFFs and asserts the fill comes back as
NaN while valid data survives, alongside targeted tests for each case above.
Reverting any of these fixes fails the corresponding test.
Mohammad Kiani
FIX stacked tif_path input and Geoeye band order for issue #66
c78d1e0 
tif_path was exempt from the single-band check so Geoeye could read
stacks through it. The filters given a stack either passed it through (a
3-band uint8 stack came back from Gaussian and Mean as a 3-channel
"filtered band") or crashed inside plt.imread with a PIL error that said
nothing about the input. Every band path is now a single band; a tool
that selects a band from a stack opts in with _multiband_tif (Geoeye).

Geoeye also sliced the wrong axis on band-interleaved stacks.
scikit-image returns those band-first (except at 3 or 4 bands), so
[:, :, level] read the width as the band count and returned a
(bands, height) slice without error. The reviewer's stacking recipe on
#66 builds exactly that file. The loader now reads the band count and
grid from the raster profile and hands Geoeye (height, width, bands).
Mohammad Kiani
FIX import-tools docs to describe the normalisation the code performs
8f0acc5 
The Geoeye section said the tool reads the sensor bit depth and divides
by 2^b - 1 via img_as_float. It does neither: it applies one min-max
stretch over the whole stack, so a single band spans only part of
[0, 1] and its scale depends on the other bands in the file.

The Landsat8 section said the raw mode returns uint16; it returns
float64 with nodata as NaN. The "rgb" and "infrared" modes stretch each
band on its own, which discards the brightness relationships between
bands. Any other exportType falls back to the raw stack.
@mhdned
mhdned merged commit 6e059ca into main Oct 17, 2026
18 checks passed
@mhdned mhdned changed the title FIX import-tools docs to describe the normalisation the code performs The Geoeye section said the tool reads the sensor bit depth and divides by 2^b - 1 via img_as_float. It does neither: it applies one min-max stretch over the whole stack, so a single band spans only part of [0, 1] and its scale depends on the other bands in the file. The Landsat8 section said the raw mode returns uint16; it returns float64 with nodata as NaN. The "rgb" and "infrared" modes stretch each band on its own, which discards the brightness relationships between bands. Any other exportType falls back to the raw stack. FIX import-tools docs to describe the normalisation the code performs Oct 17, 2026
mhdned added a commit that referenced this pull request Oct 17, 2026
@mhdned
Merge pull request #71 from FEZtool-team/main
ff290e9 
Merge pull request #70 from FEZtool-team/hotfix/nodata-safe-calculators
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aradfarahani aradfarahani aradfarahani approved these changes
@mkiani12 mkiani12 Awaiting requested review from mkiani12

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mhdned @aradfarahani