Repository navigation
ADD end-to-end smoke tests for every calculator, and fix what they found (#46) - #60
Merged
Merged
Conversation
Issue #46 reported three public methods that crash on ordinary use. All three are already fixed on main, and this change addresses the fourth section of that issue - the reason the suite could not see them. The suite covers 89% of the code, yet all three defects lived in tested code. Each slipped through for its own reason, but the shape was the same: the tests asserted that the plumbing got called, not that the code ran. The spectral profile test passed a MagicMock() as ax, which accepts ax.figure(...) and ax.xlabel(...) without complaint. The PCA test called process() on the line before the broken guard. The MagDir fixture chose values where no band difference is ever zero, so the one case that occurs in every real image pair was the one case not covered. Add tests/smoke_test.py: real synthetic georeferenced GeoTIFFs on disk, every calculator built through its public constructor, execute() and histogram_export() and to_raster() called for real, asserting a file appears. Nothing is patched. A table-driven parametrisation covers all 37 calculators, and a guard test fails if a newly exported calculator has no smoke case, so the coverage cannot quietly rot. Plus four targeted tests naming the three original defects. Running it found four further bugs the mocked tests were blind to, two of them regressions introduced with the nrows/ncols forwarding: * MosaicCalculator.execute() was broken outright. BaseTool.execute() forwards nrows/ncols to _export_file(), but MosaicCalculator overrides _export_file and was never given those parameters, so every call raised TypeError. * PCACalculator._export_file() bypasses BaseTool._export_file and so repeated neither the output-directory creation nor the tool-name fallback for filename_prefix. Since that argument now defaults to None it was interpolated literally, so PCA wrote files named "None_<uuid>.png" into a directory it had not created. * GeoeyeCalculator._validate() indexes shape[2] to count bands, raising IndexError: tuple index out of range on a single-band raster rather than saying it needs a multispectral stack. * MosaicCalculator.process() never assigns self._output; the merged raster is written directly by _export_file, which sets _output to a file path rather than an array. to_raster() therefore cannot work for this tool. Left alone and documented in the skip reason, since changing it alters public behaviour and three existing tests assert the current semantics. The harness was verified by reintroducing each of the three original defects and confirming the corresponding test fails, then reverting. Without that check these tests would only be assumed to work, which is the complaint the issue makes about the existing suite. 521 passed on Python 3.12, 520 on 3.11 and 3.13, all headless. The smoke suite adds about seven seconds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses issue #46.
The three reported bugs are already fixed on
mainVerified before writing anything, by running the reproducers:
SpectralProfileCalculator.histogram_export()ax.figure(...), usesax.set_xlabel/set_ylabelxaxis/yaxisboth length 6PCACalculator.histogram_export()if self._output is Noneexecute()firstMagDirCalculatorties0grepconfirms no otherhasattr(self, "_output")survives, as you suggested checking.So this PR is about your fourth section
That was the part worth acting on. 89% coverage, and all three defects lived in covered code — because the tests asserted that the plumbing got called rather than that the code ran. Your diagnosis of each was exact: the
MagicMock()asaxacceptsax.figure(...)happily; the PCA test calledprocess()on the line before the broken guard; the MagDir fixture used values where no difference is ever zero.tests/smoke_test.pyis the counterweight, built roughly as you described — real synthetic georeferenced GeoTIFFs on disk, every calculator constructed through its public API,execute()/histogram_export()/to_raster()called for real, assert a file appears. Nothing patched.It came out a bit larger than the ~20 lines you estimated, mostly because the 37 constructors are irregular (
channel,exportType,level,time,operation,selecte, kernel sizes,tif_paths). Keeping it table-driven contains that, and a guard test fails if a newly exported calculator has no smoke case, so the coverage can't quietly rot. GLCM gets an 8×8 raster of its own so its per-pixel loop stays in milliseconds.Notably
SVMCalculatoris in there — it's smoke-testable at all only because the non-interactivetraining_samplespath from #39 exists.It immediately found four more bugs
Which is the argument for the approach, so worth listing. Two are regressions from the
nrows/ncolsforwarding added for the docs/API issue:MosaicCalculator.execute()was broken outright.BaseTool.execute()forwardsnrows/ncolsto_export_file(), butMosaicCalculatoroverrides_export_fileand never got those parameters — every call raisedTypeError: takes from 2 to 11 positional arguments but 13 were given. Fixed.PCA wrote files called
None_<uuid>.png.PCACalculator._export_file()bypassesBaseTool._export_file, so it repeated neither the output-directory creation nor the tool-name fallback forfilename_prefix. Since that argument now defaults toNone, it was interpolated literally — into a directory that didn't exist, so it raisedFileNotFoundErrortoo. Fixed.GeoeyeCalculatoron a single-band raster raisedIndexError: tuple index out of rangefromshape[2], rather than saying it needs a multispectral stack. Now a clearValueError. (My first smoke case fed it the wrong input — but the error should say so.)MosaicCalculator.process()never assignsself._output. The merged raster is written directly by_export_file, which sets_outputto a file path rather than an array, soto_raster()can't work for this tool and the documentedBaseToollifecycle doesn't hold. Left alone deliberately — fixing it changes public behaviour and three existing tests assert the current semantics. It's documented in the smoke test's skip reason; happy to take it as a separate issue if you think it should change.Proving the tests actually bite
The complaint in your issue is that tests can look like they cover something without measuring it, so it seemed wrong to just assert these are better. I reintroduced each of the three original defects and confirmed the corresponding tests fail:
Both the generic harness and the targeted tests fire on each. Then reverted.
Results
521 passed on Python 3.12; 520 on 3.11 and 3.13; all runs headless with
DISPLAY/WAYLAND_DISPLAYunset. The smoke suite adds ~7 seconds. The 25 skips are the 24 calculators without ahistogram_export, plus the documented Mosaicto_rastercase.Not included
GLCM's per-pixel cost, which you mention alongside the MagDir vectorization. MagDir is done, but GLCM is a genuine windowed co-occurrence computation rather than something four
np.wherecalls replace, and it got ~4× more expensive when angle-averaging was added for #38. It deserves its own PR with before/after timings and numerical-equivalence tests, rather than being folded into a correctness change.