Skip to content

fix(classifier): resize and centre-crop each image as its model was evaluated - #719

Draft
Yukkta-Seelam wants to merge 6 commits into
developfrom
feat/classifier-crop-host
Draft

Yukkta-Seelam wants to merge 6 commits into
developfrom
feat/classifier-crop-host

Conversation

@Yukkta-Seelam

@Yukkta-Seelam Yukkta-Seelam commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The image classification explorer squeezed every image to 224x224, which distorts its shape. The models were evaluated differently: the shorter side is scaled to 256, then the centred 224x224 window is cut out. This PR prepares each image the way its model was evaluated, in both the C++ and Python entrypoints.

This is the interim, application-side version. Neat's ResizeMode::Crop cannot do it yet: it fails on Modalix (sima-neat/core#1320), and it cannot resize to 256 before cropping. Once the Core part of #1320 lands, the app can hand the crop to Neat; see the follow-up below.

Changes

  • Resize and centre crop per model. Each profile gains two settings:
    • resize_shorter (default 256 for a 224 input): the size the shorter side is scaled to before the centred input_width x input_height window is cut.
    • interpolation (bilinear or bicubic): the filter the model was evaluated with. ResNet-18 and ResNeXt-101 64x4d use bilinear; ResNet-50 and DenseNet-121 use bicubic, as in their Model Registry recipes.
  • Shrinking uses area averaging. The registry evaluated these models with Pillow, whose bilinear and bicubic filters average over the source when they shrink. OpenCV's do not, so the app shrinks with INTER_AREA and uses the configured filter only to enlarge small images. On 1,000 ImageNet validation images, this brought the app's input to a mean of about 1.1 pixel levels (out of 255) from Pillow's, against 2.3 to 3.0 with OpenCV's own bilinear and bicubic. Pillow's bilinear and bicubic differ from each other by 1.55.
  • The crop offset rounds like torchvision. It rounds to the nearest pixel, with ties to even, as torchvision's CenterCrop and the ResNet-18 and DenseNet-121 recipes do. The ResNet-50 recipe truncates, so on images with an odd margin the two differ by one pixel.
  • Model Registry packages. The resize settings describe how the registry built and evaluated each model, so the app now loads the registry's w16a16 builds of resnet_50, resnet_18, densenet_121 and resnext_101_64x4d. The config, README, tests/test-scope.yaml (ref: main) and both e2e suites are updated.
  • The README extracts each download. sima-cli models download saves models/<id>/w16a16/<id>_w16a16_default.zip, which wraps the <id>_w16a16_mpk.tar.gz the config names, so the README unzips each package in place after downloading it.
  • EfficientNet-B0 replaced by ResNeXt-101 64x4d. EfficientNet-B0 has no Model Registry recipe, so its filter would be a guess on a Model Zoo package with unrecorded preprocessing. ResNeXt-101 64x4d takes its place: it is the most accurate classifier with a w16a16 package on registry main, and its recipe is the same as ResNet-18's, so it needs only a new profile. The app still compares four models.

Evidence

Modalix DevKit, Core 0.6.1+develop.f70653154707, branch head 4729170f built natively on the board. ResNet-50, ResNet-18 and ResNeXt-101 64x4d came from the registry's production main catalog, downloaded with the README's commands. DenseNet-121 came from its PR branch build (516-add-densenet-imagenet-classifiers), because neither develop nor main has a completed build of it (see Open items).

Check Result
C++ unit and e2e (ctest -R image-classification-explorer) PASS with all 4 models
Python unit and e2e (pytest) 262 passed, 0 skipped
C++ and Python reports on the goldfish sample Same top-1 class and probability for all 4 models
scripts/validate_readmes.py, scripts/generate_catalog.py, tests/scripts/test_scope_contract.py PASS
SDK container build (./build.sh) PASS, no warnings, at 9d60d9f9; not rerun for the config-only 4729170f

Top-1 accuracy (top-5 in brackets) on the first 2,000 ImageNet 2012 validation images, same board, same models:

Model Before (squeeze) After (crop) Registry, 50,000 images
resnet_50 w16a16 79.05% (94.30%) 80.05% (94.90%) 80.22%
resnet_18 w16a16 68.20% (87.85%) 69.90% (88.55%) 69.78%
densenet_121 w16a16 73.50% (91.35%) 75.55% (92.25%) 74.74%
resnext_101_64x4d w16a16 not in the old app 82.95% (96.70%) 82.97%

The first three rows were measured at 9d60d9f9 with the registry's develop packages; the ResNeXt row at 4729170f with its main package. Between those commits only the README and the ResNeXt profile changed, not the app code.

Open items

  • CI e2e waits for DenseNet-121. main now has ResNet-50, ResNet-18 and ResNeXt-101 64x4d. It has no DenseNet-121 build, so the e2e jobs fail with "No completed model build found" for it.
  • DenseNet-121 needs a build. Every develop compile of densenet_121 lost its runner after sima-neat/models#545 merged, and sima-neat/models#793 then made it build on demand. It needs an orchestrator dispatch and promotions to develop and main.
  • Some DevKit CI shards fail before any model test with "Developer Portal model access is not provisioned on this runner". That is runner setup, not this change.
  • Not verified: the install check. sima-cli neat install apps@feat/classifier-crop-host and the installed README walkthrough wait until the models are in main.

Follow-up

Move the crop into Neat once the Core part of sima-neat/core#1320 lands. sima-neat/kernels#115 added the lower-level resize window, but it supports bilinear and antialiased bilinear, not bicubic. So Neat can take over the crop for ResNet-18 only, until bicubic windows are supported too.

The explorer squeezed every image to 224x224 with INTER_AREA, so the models
saw distorted shapes and borders they were never evaluated on. ImageNet
classifiers are evaluated by scaling the shorter side to 256 and cutting the
centred 224x224 window, with a model-specific filter.

Each profile now sets resize_shorter (default 256 for a 224 input) and
interpolation (bilinear or bicubic). Both entrypoints resize and crop on the
host with the same integer geometry, and average large images down to twice
the target first so OpenCV's filters do not alias. Neat's ResizeMode::Crop
cannot do this yet (sima-neat/core#1320), so Neat receives an image that
already has the model's input size.
The resize filter and pre-crop size each profile now sets come from the
Model Registry recipes, so they only describe how the model was evaluated
when the app runs the registry's own package. Switch resnet_50, resnet_18
and densenet_121 to their w16a16 registry builds, the most accurate
variant all three publish, in the config, README, test scope and both
e2e suites.

efficientnet_b0 has no Model Registry recipe, so its profile and test
entries are removed rather than kept on a Model Zoo package whose
preprocessing is unknown.
The Model Registry evaluates these models with Pillow, whose bilinear
and bicubic filters widen with the scale factor when shrinking. OpenCV's
filters do not, so the host crop drifted from the registry's input by
about 3 pixel levels on average across ImageNet validation images, more
than the 1.55 between Pillow's own two filters. Shrinking with OpenCV's
area averaging brings that to about 1.1 for the bilinear and bicubic
recipes alike; the profile's filter still enlarges small images.

The crop offset now rounds to the nearest pixel with ties to even, as
torchvision's CenterCrop and the resnet_18 and densenet_121 recipes do,
instead of truncating, which moved the window by one pixel on images
with an odd margin. The resnet_50 recipe truncates; it keeps a one-pixel
shift on those images (mean gap 2.2).
Measured on a Modalix DevKit (Core 0.6.1+develop.f7065315) with the
w16a16 registry builds; the C++ and Python reports were identical.
sima-cli models download saves each package as
models/<id>/<variant>/<id>_<variant>_default.zip, which wraps the
<id>_<variant>_mpk.tar.gz the config names. The README said the tar.gz
landed there directly, so a customer following it hit a missing model
file. Unzip each package in place after downloading it.
EfficientNet-B0 has no Model Registry recipe, so the explorer compared
only three models. ResNeXt-101 64x4d is the most accurate classifier
with a w16a16 package on registry main (expected top-1 0.8297), and its
registry recipe is the same as ResNet-18's: bilinear resize of the
shorter side to 256, centre crop to 224, ImageNet mean and std. It
therefore needs only a new profile, with no change to either entrypoint.

On the DevKit (Core 0.6.1+develop.f70653154707) the app's centre-crop
path gives top-1 0.8295 and top-5 0.9670 on the 2000-image ImageNet
validation subset. The bundled goldfish sample is class 1 for all four
models in both languages; ResNeXt's probability is lower (0.4966).

This branch was successfully deployed

3 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant