Repository navigation
fix(classifier): resize and centre-crop each image as its model was evaluated - #719
Draft
Yukkta-Seelam wants to merge 6 commits into
Draft
Yukkta-Seelam wants to merge 6 commits into
Yukkta-Seelam wants to merge 6 commits into
Conversation
The explorer squeezed every image to 224x224 with INTER_AREA, so the models saw distorted shapes and borders they were never evaluated on. ImageNet classifiers are evaluated by scaling the shorter side to 256 and cutting the centred 224x224 window, with a model-specific filter. Each profile now sets resize_shorter (default 256 for a 224 input) and interpolation (bilinear or bicubic). Both entrypoints resize and crop on the host with the same integer geometry, and average large images down to twice the target first so OpenCV's filters do not alias. Neat's ResizeMode::Crop cannot do this yet (sima-neat/core#1320), so Neat receives an image that already has the model's input size.
The resize filter and pre-crop size each profile now sets come from the Model Registry recipes, so they only describe how the model was evaluated when the app runs the registry's own package. Switch resnet_50, resnet_18 and densenet_121 to their w16a16 registry builds, the most accurate variant all three publish, in the config, README, test scope and both e2e suites. efficientnet_b0 has no Model Registry recipe, so its profile and test entries are removed rather than kept on a Model Zoo package whose preprocessing is unknown.
The Model Registry evaluates these models with Pillow, whose bilinear and bicubic filters widen with the scale factor when shrinking. OpenCV's filters do not, so the host crop drifted from the registry's input by about 3 pixel levels on average across ImageNet validation images, more than the 1.55 between Pillow's own two filters. Shrinking with OpenCV's area averaging brings that to about 1.1 for the bilinear and bicubic recipes alike; the profile's filter still enlarges small images. The crop offset now rounds to the nearest pixel with ties to even, as torchvision's CenterCrop and the resnet_18 and densenet_121 recipes do, instead of truncating, which moved the window by one pixel on images with an odd margin. The resnet_50 recipe truncates; it keeps a one-pixel shift on those images (mean gap 2.2).
Measured on a Modalix DevKit (Core 0.6.1+develop.f7065315) with the w16a16 registry builds; the C++ and Python reports were identical.
sima-cli models download saves each package as models/<id>/<variant>/<id>_<variant>_default.zip, which wraps the <id>_<variant>_mpk.tar.gz the config names. The README said the tar.gz landed there directly, so a customer following it hit a missing model file. Unzip each package in place after downloading it.
EfficientNet-B0 has no Model Registry recipe, so the explorer compared only three models. ResNeXt-101 64x4d is the most accurate classifier with a w16a16 package on registry main (expected top-1 0.8297), and its registry recipe is the same as ResNet-18's: bilinear resize of the shorter side to 256, centre crop to 224, ImageNet mean and std. It therefore needs only a new profile, with no change to either entrypoint. On the DevKit (Core 0.6.1+develop.f70653154707) the app's centre-crop path gives top-1 0.8295 and top-5 0.9670 on the 2000-image ImageNet validation subset. The bundled goldfish sample is class 1 for all four models in both languages; ResNeXt's probability is lower (0.4966).
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The image classification explorer squeezed every image to 224x224, which distorts its shape. The models were evaluated differently: the shorter side is scaled to 256, then the centred 224x224 window is cut out. This PR prepares each image the way its model was evaluated, in both the C++ and Python entrypoints.
This is the interim, application-side version. Neat's
ResizeMode::Cropcannot do it yet: it fails on Modalix (sima-neat/core#1320), and it cannot resize to 256 before cropping. Once the Core part of #1320 lands, the app can hand the crop to Neat; see the follow-up below.Changes
resize_shorter(default 256 for a 224 input): the size the shorter side is scaled to before the centredinput_widthxinput_heightwindow is cut.interpolation(bilinearorbicubic): the filter the model was evaluated with. ResNet-18 and ResNeXt-101 64x4d use bilinear; ResNet-50 and DenseNet-121 use bicubic, as in their Model Registry recipes.INTER_AREAand uses the configured filter only to enlarge small images. On 1,000 ImageNet validation images, this brought the app's input to a mean of about 1.1 pixel levels (out of 255) from Pillow's, against 2.3 to 3.0 with OpenCV's own bilinear and bicubic. Pillow's bilinear and bicubic differ from each other by 1.55.CenterCropand the ResNet-18 and DenseNet-121 recipes do. The ResNet-50 recipe truncates, so on images with an odd margin the two differ by one pixel.w16a16builds ofresnet_50,resnet_18,densenet_121andresnext_101_64x4d. The config, README,tests/test-scope.yaml(ref: main) and both e2e suites are updated.sima-cli models downloadsavesmodels/<id>/w16a16/<id>_w16a16_default.zip, which wraps the<id>_w16a16_mpk.tar.gzthe config names, so the README unzips each package in place after downloading it.w16a16package on registrymain, and its recipe is the same as ResNet-18's, so it needs only a new profile. The app still compares four models.Evidence
Modalix DevKit, Core
0.6.1+develop.f70653154707, branch head4729170fbuilt natively on the board. ResNet-50, ResNet-18 and ResNeXt-101 64x4d came from the registry's productionmaincatalog, downloaded with the README's commands. DenseNet-121 came from its PR branch build (516-add-densenet-imagenet-classifiers), because neitherdevelopnormainhas a completed build of it (see Open items).ctest -R image-classification-explorer)scripts/validate_readmes.py,scripts/generate_catalog.py,tests/scripts/test_scope_contract.py./build.sh)9d60d9f9; not rerun for the config-only4729170fTop-1 accuracy (top-5 in brackets) on the first 2,000 ImageNet 2012 validation images, same board, same models:
The first three rows were measured at
9d60d9f9with the registry'sdeveloppackages; the ResNeXt row at4729170fwith itsmainpackage. Between those commits only the README and the ResNeXt profile changed, not the app code.Open items
mainnow has ResNet-50, ResNet-18 and ResNeXt-101 64x4d. It has no DenseNet-121 build, so the e2e jobs fail with "No completed model build found" for it.developcompile ofdensenet_121lost its runner after sima-neat/models#545 merged, and sima-neat/models#793 then made it build on demand. It needs an orchestrator dispatch and promotions todevelopandmain.sima-cli neat install apps@feat/classifier-crop-hostand the installed README walkthrough wait until the models are inmain.Follow-up
Move the crop into Neat once the Core part of sima-neat/core#1320 lands. sima-neat/kernels#115 added the lower-level resize window, but it supports bilinear and antialiased bilinear, not bicubic. So Neat can take over the crop for ResNet-18 only, until bicubic windows are supported too.