Skip to content

ADD end-to-end smoke tests for every calculator, and fix what they found (#46) - #60

Merged
mkiani12 merged 1 commit into
mainfrom
feat/smoke-tests-for-every-calculator
Aug 28, 2026
Merged

mkiani12 merged 1 commit into
mainfrom
feat/smoke-tests-for-every-calculator

Conversation

@mkiani12

Copy link
Copy Markdown
Collaborator

Addresses issue #46.

The three reported bugs are already fixed on main

Verified before writing anything, by running the reproducers:

state check
SpectralProfileCalculator.histogram_export() fixed — no ax.figure(...), uses ax.set_xlabel/set_ylabel writes a real PNG; xaxis/yaxis both length 6
PCACalculator.histogram_export() fixed — if self._output is None works on a fresh instance without execute() first
MagDirCalculator ties fixed, and vectorized identical dates → magnitude all zero, direction all 0

grep confirms no other hasattr(self, "_output") survives, as you suggested checking.

So this PR is about your fourth section

That was the part worth acting on. 89% coverage, and all three defects lived in covered code — because the tests asserted that the plumbing got called rather than that the code ran. Your diagnosis of each was exact: the MagicMock() as ax accepts ax.figure(...) happily; the PCA test called process() on the line before the broken guard; the MagDir fixture used values where no difference is ever zero.

tests/smoke_test.py is the counterweight, built roughly as you described — real synthetic georeferenced GeoTIFFs on disk, every calculator constructed through its public API, execute() / histogram_export() / to_raster() called for real, assert a file appears. Nothing patched.

It came out a bit larger than the ~20 lines you estimated, mostly because the 37 constructors are irregular (channel, exportType, level, time, operation, selecte, kernel sizes, tif_paths). Keeping it table-driven contains that, and a guard test fails if a newly exported calculator has no smoke case, so the coverage can't quietly rot. GLCM gets an 8×8 raster of its own so its per-pixel loop stays in milliseconds.

Notably SVMCalculator is in there — it's smoke-testable at all only because the non-interactive training_samples path from #39 exists.

It immediately found four more bugs

Which is the argument for the approach, so worth listing. Two are regressions from the nrows/ncols forwarding added for the docs/API issue:

MosaicCalculator.execute() was broken outright. BaseTool.execute() forwards nrows/ncols to _export_file(), but MosaicCalculator overrides _export_file and never got those parameters — every call raised TypeError: takes from 2 to 11 positional arguments but 13 were given. Fixed.

PCA wrote files called None_<uuid>.png. PCACalculator._export_file() bypasses BaseTool._export_file, so it repeated neither the output-directory creation nor the tool-name fallback for filename_prefix. Since that argument now defaults to None, it was interpolated literally — into a directory that didn't exist, so it raised FileNotFoundError too. Fixed.

GeoeyeCalculator on a single-band raster raised IndexError: tuple index out of range from shape[2], rather than saying it needs a multispectral stack. Now a clear ValueError. (My first smoke case fed it the wrong input — but the error should say so.)

MosaicCalculator.process() never assigns self._output. The merged raster is written directly by _export_file, which sets _output to a file path rather than an array, so to_raster() can't work for this tool and the documented BaseTool lifecycle doesn't hold. Left alone deliberately — fixing it changes public behaviour and three existing tests assert the current semantics. It's documented in the smoke test's skip reason; happy to take it as a separate issue if you think it should change.

Proving the tests actually bite

The complaint in your issue is that tests can look like they cover something without measuring it, so it seemed wrong to just assert these are better. I reintroduced each of the three original defects and confirmed the corresponding tests fail:

mutated: spectral profile ax.figure(...) restored
mutated: PCA hasattr guard restored
mutated: MagDir zero-case now inherits class 1

FAILED test_histogram_export_writes_a_png[SpectralProfileCalculator]
FAILED test_spectral_profile_histogram_export_runs
FAILED test_histogram_export_writes_a_png[PCACalculator]
FAILED test_pca_histogram_export_works_without_execute_first
FAILED test_magdir_handles_pixels_that_did_not_change

Both the generic harness and the targeted tests fire on each. Then reverted.

Results

521 passed on Python 3.12; 520 on 3.11 and 3.13; all runs headless with DISPLAY/WAYLAND_DISPLAY unset. The smoke suite adds ~7 seconds. The 25 skips are the 24 calculators without a histogram_export, plus the documented Mosaic to_raster case.

Not included

GLCM's per-pixel cost, which you mention alongside the MagDir vectorization. MagDir is done, but GLCM is a genuine windowed co-occurrence computation rather than something four np.where calls replace, and it got ~4× more expensive when angle-averaging was added for #38. It deserves its own PR with before/after timings and numerical-equivalence tests, rather than being folded into a correctness change.

Issue #46 reported three public methods that crash on ordinary use. All three
are already fixed on main, and this change addresses the fourth section of that
issue - the reason the suite could not see them.

The suite covers 89% of the code, yet all three defects lived in tested code.
Each slipped through for its own reason, but the shape was the same: the tests
asserted that the plumbing got called, not that the code ran. The spectral
profile test passed a MagicMock() as ax, which accepts ax.figure(...) and
ax.xlabel(...) without complaint. The PCA test called process() on the line
before the broken guard. The MagDir fixture chose values where no band
difference is ever zero, so the one case that occurs in every real image pair
was the one case not covered.

Add tests/smoke_test.py: real synthetic georeferenced GeoTIFFs on disk, every
calculator built through its public constructor, execute() and
histogram_export() and to_raster() called for real, asserting a file appears.
Nothing is patched. A table-driven parametrisation covers all 37 calculators,
and a guard test fails if a newly exported calculator has no smoke case, so the
coverage cannot quietly rot. Plus four targeted tests naming the three original
defects.

Running it found four further bugs the mocked tests were blind to, two of them
regressions introduced with the nrows/ncols forwarding:

  * MosaicCalculator.execute() was broken outright. BaseTool.execute() forwards
    nrows/ncols to _export_file(), but MosaicCalculator overrides _export_file
    and was never given those parameters, so every call raised TypeError.

  * PCACalculator._export_file() bypasses BaseTool._export_file and so repeated
    neither the output-directory creation nor the tool-name fallback for
    filename_prefix. Since that argument now defaults to None it was
    interpolated literally, so PCA wrote files named "None_<uuid>.png" into a
    directory it had not created.

  * GeoeyeCalculator._validate() indexes shape[2] to count bands, raising
    IndexError: tuple index out of range on a single-band raster rather than
    saying it needs a multispectral stack.

  * MosaicCalculator.process() never assigns self._output; the merged raster is
    written directly by _export_file, which sets _output to a file path rather
    than an array. to_raster() therefore cannot work for this tool. Left alone
    and documented in the skip reason, since changing it alters public
    behaviour and three existing tests assert the current semantics.

The harness was verified by reintroducing each of the three original defects
and confirming the corresponding test fails, then reverting. Without that check
these tests would only be assumed to work, which is the complaint the issue
makes about the existing suite.

521 passed on Python 3.12, 520 on 3.11 and 3.13, all headless. The smoke suite
adds about seven seconds.
@mkiani12
mkiani12 merged commit 4980f23 into main Aug 28, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant