Skip to content

feat(boxdecode): Segment Anything masks for MobileSAM, EdgeTAM, EfficientSAM and EfficientSAM3 - #1355

Open
manuel-roldan wants to merge 13 commits into
developfrom
feat/boxdecode-sam-mask
Open

manuel-roldan wants to merge 13 commits into
developfrom
feat/boxdecode-sam-mask

Conversation

@manuel-roldan

@manuel-roldan manuel-roldan commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Refs sima-neat/models#390

Summary

This PR decodes Segment Anything models through BoxDecode. There is one decode type per model. A run returns the chosen mask at the original image size, with its box and predicted IoU, in a versioned masks payload. decode_segmentation reads that payload.

MobileSAM (sima-neat/models#390) is the issue this PR serves. The other model PRs it was checked against are:

  • EdgeTAM: sima-neat/models#823
  • EfficientSAM: sima-neat/models#503 and sima-neat/models#538
  • EfficientSAM3: sima-neat/models#232

Depends on sima-neat/kernels#119. That PR adds the Kernels BoxDecode profiles sam_mask and sam_concept. Core builds the named request; Kernels admits it, selects the mask, resamples it and applies the sign test. See Dependency and CI below.

The PR has three commits, meant to be reviewed in order:

  1. feat(preproc): top-left letterbox placement, and the ABI bump to 10.
  2. feat(boxdecode): the Segment Anything decode types, multimask_output and boxdecode_raw_outputs, with tests.
  3. docs(boxdecode): reference docs, release notes and the four translations.

Decode types

Model BoxDecodeType (token) Preprocess of the Model that sees the image Decoder options
MobileSAM MobileSam = 28 (mobilesam) Encoder: resize.mode=Letterbox, resize.placement=TopLeft, 1024×1024, pad_value=116, preset=ImageNet multimask_output
EdgeTAM EdgeTam = 29 (edgetam) Image encoder: resize.mode=Stretch, 1024×1024, preset=ImageNet multimask_output; boxdecode_raw_outputs for tracking
EfficientSAM-T / -S EfficientSam = 30 (efficientsam) Encoder: resize.mode=Stretch, 1024×1024, values 0–1 only (the encoder normalizes itself) leave multimask_output true
EfficientSAM3 prompt decoder EfficientSam3 = 31 (efficientsam3) Image encoder: resize.mode=Stretch, 1008×1008, mean and stddev 0.5 multimask_output
EfficientSAM3 concept image model EfficientSam3 = 31 The image model itself, on the tensor route: the application stretches to 1008, scales to [−1, 1] and attaches semantic.preprocess top_k, score_threshold
  • Aliases: mobile-sam, edge-tam, efficient-sam, efficient-sam3.
  • Python: pyneat.BoxDecodeType.MobileSam, EdgeTam, EfficientSam, EfficientSam3.

Pseudocode for one prompt-decoder run:

heads   = bind by position and shape:
            logits [h,w,C]  (h,w > 1; C in {1,3,4})
            iou    [C]      (next output with C values)
            object [1]      (later single value, EdgeTAM only)
c       = select(decode_type, C, multimask_output, iou)        # see the mapping below
canvas  = semantic.preprocess of the decoder inputs            # S×S, top-left letterbox or stretch
mask    = resample(logits[:,:,c] -> original H×W through the canvas geometry) > 0
if object is present and object <= 0: mask = empty             # upstream object-score gate
record  = { box: tight box of mask, score: iou[c], class: c }  # count is always 1
wire    = SegmentationMasksHeaderV1 | record | uint8 mask [H,W]

The concept image model is different. It binds detections [Q,6] and logits [h,w,Q]. It keeps up to top_k queries above score_threshold, in descending score order with ties broken by query index. It returns each kept query's box from the detection head and that query's own mask.

Model::Options::multimask_output (Python ModelOptions.multimask_output, default true)

This option mirrors upstream predict(multimask_output=...).

decode_type true false
MobileSam (C=4) best predicted IoU of masks 1–3 mask 0
EdgeTam (C=4) best predicted IoU of masks 1–3 SAM2 stability rule: mask 0 if stable (δ 0.05, threshold 0.98), else best of 1–3
EfficientSam (C=3) best predicted IoU of masks 0–2 rejected: "EfficientSAM exports no single-mask token"
EfficientSam3, C=4 / C=3 best of 1–3 / best of 0–2 stability rule / rejected
EfficientSam3 concept up to top_k queries rejected (no multimask choice)
  • A one-channel export uses its only mask with either value.
  • Other decode types reject false when the BoxDecode stage is built, as pose_classes already does.
  • A decoder whose channel count does not match the named model is rejected, for example a three-mask decoder declared as MobileSam.
  • EdgeTAM's object-score gate turns on automatically when the decoder exports that output.

Model::Options::boxdecode_raw_outputs (default false)

When this is true, the route returns the BoxDecode tensor first. After it come the model's own outputs, exactly as the same Model returns them without BoxDecode: the same names, dtypes and shapes, from the same inference run. EdgeTAM tracking needs this, because the memory encoder and bank take the chosen plane's low-resolution logits and the object pointer.

How the raw copies are made:

  • Each raw output is one extra member of the BoxDecode stage. BF16 heads are cast, INT8 heads dequantized, and FP32 heads copied.
  • These members read only the stage inputs, so the decode never waits on them.
  • The detection tag stays on the first tensor.

The default route is unchanged.

Top-left letterbox placement

ResizeSpec::placement takes LetterboxPlacement::Center (the default) or TopLeft. The Preproc frame adapter already implements top-left placement and publishes the placed rectangle and its affine.

  • Validation: the planner rejects TopLeft unless the resize is an enabled letterbox.
  • Rounding: a Model top-left letterbox rounds the shorter side to the nearest pixel, as SAM does (640×427 → 1024×683, pad 341). Centered letterboxes and the Preproc node's own placements still round up, as before.
  • Serialization: graph save/load and the stage cache identity record a non-default placement. Center keeps its existing serialized form and identity.
  • Detection profiles: these switch to the explicit model-to-original transform. A cold graph is seeded from the checked Preproc policy, the same way Crop is.
  • Pose, segmentation and proposal profiles: these reject a non-centered letterbox rather than return shifted boxes.

Geometry rules

  • The canvas, the original image size and the resize come only from the decoder inputs' semantic.preprocess. Pass the encoder outputs with .cvu().
  • MobileSAM requires a top-left letterbox. The other models require a stretch.
  • A mismatch fails at graph build, and the error names the encoder setting to change. Example: "MobileSAM needs the image preprocessed with resize.mode=Letterbox and resize.placement=TopLeft on the Model that preprocesses it".
  • An explicit transform, a crop or missing metadata is rejected.
  • A runner serves one original image size, because the masks have that size. Model::run builds a new runner when the size changes; a streaming graph rejects a frame of another size.

ABI

The ABI moves from 9 to 10 (libsima_neat.so.10).

What changed:

  • ResizeSpec gains placement.
  • Model::Options gains boxdecode_raw_outputs and multimask_output.

Why this is a bump and not a fold into 9:

What else this PR touches for the ABI:

  • deps/manifest.json changes only abi-version. The dependency pins and policies are untouched.
  • The 0.6.1 release notes and the pages that state the current ABI (graph, Preproc, BoxDecode decode types) now say 10.

No public signature is renamed, removed or reordered. The rest is additive:

  • the LetterboxPlacement enum;
  • BoxDecodeType values 28–31;
  • SegmentationMasksHeaderV1 in DetectionTypes.h;
  • decode_segmentation / decode_bbox recognizing the V1 payload by its magic.

Against develop, YOLO-seg requests and the headerless 160×160 segmentation payload are byte-identical, and centered letterboxes keep their mapping.

Dependency and CI

The Kernels profiles live in sima-neat/kernels#119, which is not yet in the dependency set Core CI resolves.

What is independent of the dependency:

  • Core builds and links against the current set; the SAM request is generic, sent through check_kernel.
  • The Scheduler and LLiMa policies are snap, so once feat: sync NEAT install between eLxr SDK and paired DevKit #119 is merged and its Runner/Scheduler packages are published, CI picks them up with no change to deps/manifest.json.

What Core CI can exercise now:

  • the build, format, include, header, naming and artifact checks, and the i18n checks;
  • every pure-contract test:
    • extractor binding for the five model/decoder shapes;
    • stage semantics;
    • the multimask_output mapping and every rejection;
    • geometry, benchmark geometry, stage identity and graph save/load;
    • the masks wire, payload decoding, publication and Python.
  • unit_boxdecode_kernel_recipe_test --sam-core-contract and --boxdecode-raw-outputs-core-contract, which fail before admission.

What it cannot exercise until #119 is published:

  • the default run of unit_boxdecode_kernel_recipe_test. Its SAM and raw-output admission cases need the sam_mask and sam_concept profiles, and they fail at Kernels admission against the current set;
  • the device runs in CI. They were run by hand on a Modalix DevKit (see Test Evidence).

The "Merge readiness" section below proposes the merge order.

Change Type

  • Bug fix
  • Refactor
  • Feature
  • Docs
  • Test-only
  • Build/CI
  • Breaking change (ABI 10; no API signature change)

Risk

  • Low
  • Medium
  • High

Risk notes:

  • Shared admission path. The BoxDecode admission path is shared with the detection profiles, and placement detection adds a branch there. Centered letterboxes keep their mapping: opposite pads that differ by one pixel count as centering remainder. YOLO-seg requests are byte-identical to develop.
  • New raw-output members. These exist only when boxdecode_raw_outputs is set.
  • ABI bump. Consumers must rebuild.

Test Evidence

Commands run and outcomes:

Host (x86, Core sources under test compiled against a stubbed Scheduler client; recipes echoed, not admitted)
  SAM decoder MPK extraction: real MobileSAM INT8/BF16 decoder manifests; EdgeTAM and EfficientSAM manifests
    synthesized from the models scripts; EfficientSAM3 Core strict seed              PASS
  Named requests per model (selection, first/count, stability parameters, object-score gate, geometry)  PASS
  Rejections: MobileSAM on stretch, EdgeTAM on letterbox, EfficientSAM multimask_output=false,
    three-mask EfficientSAM3 multimask_output=false, concept multimask_output=false   rejected by name
  Unit-test mirrors: sam_core_contract, sam_recipe, boxdecode_raw_outputs, sam_mask_static_contract,
    sam_family_static_contracts, raw strided FP32 copy, detection tag                   PASS
  MobileSAM wire end to end (1280x720, 1920x1080, 640x427, empty mask)                PASS
  Python-built V1 payloads decoded by C++                                              PASS
  YOLO-seg recipes (request, wiring, wire extent) vs develop                           IDENTICAL
  Legacy 160x160 wire publication (36 cases) / payload decoding (720 cases) vs develop IDENTICAL

aarch64 cross build (GCC14), current develop dependency set: clean, no warnings in changed files
  sima_neat (libsima_neat.so.10), _pyneat_core, unit_boxdecode_kernel_recipe_test,
  unit_sima_boxdecode_static_contract_extractor_test, unit_boxdecode_geometry_test,
  unit_stage_cache_identity_test, unit_benchmark_measurement_contract_test,
  unit_model_output_spec_contract_test, unit_graph_io_contract_test, unit_boxdecode_publication_test,
  unit_detection_types_boxdecode_payloads_test, unit_input_planner_semantics_test,
  unit_preproc_contract_rules_test, unit_model_semantic_segmentation_test
  First commit alone: changed translation units compile (-fsyntax-only)

A65 link against kernels#119 HEAD 10f09ea (Kernels -> Runner -> Scheduler rebuilt on it)
  Same targets link against the Scheduler runtime that carries the sam_mask/sam_concept fields
  (A65-only: the EV74 firmware cohort check does not yet accept #119's EV sources)

Repo checks
  scripts/check_format.sh --changed-only                OK
  scripts/check_cmake_format.sh --changed-only          OK (no CMake changes)
  scripts/check_duplicate_includes.sh --changed-only    OK
  scripts/check_internal_headers.sh                     OK
  scripts/ci/check_naming_and_conflicts.sh              OK
  scripts/ci/check_artifacts.sh                         OK
  scripts/ci/check_release_policy.sh                    OK
  sima-i18n check --require-complete                    97/97 per locale (ko, ja, zh-Hant, uk)
  check-i18n-catalogs / preview navigation / developer-center links   OK

Board (Modalix DevKit) at fe24a13a with kernels#119 10f09ea; board restored and verified afterwards
  Core aarch64 unit tests                                              27/27 PASS
  Python tests                                                         89 passed, 24 skipped
  MobileSAM, EdgeTAM, EfficientSAM-T (w8a8, w16a16), EfficientSAM3 concept
    masks vs exact reference (INT8: integer codes)                     0 mismatches
    raw outputs vs the route without BoxDecode                         names, dtypes, shapes, bytes equal
  MobileSAM throughput (EV74 preprocess + MLA + BoxDecode)             20.76 FPS, p50 62.7 ms

Device details, and the defects the earlier device runs found (fixed by the appended commits), are in the board results comment: #1355 (comment)

Docs Impact

  • No docs update needed
  • Docs updated in this PR
  • Docs follow-up required (linked issue)

Docs notes:

  • reference/boxdecode_decode_types:
    • a per-model table (decode type, the encoder Model's preprocess, decoder options);
    • the multimask_output mapping and rejections;
    • what each output means, and the rules;
    • Python examples for MobileSAM, EdgeTAM tracking with raw outputs, EfficientSAM and EfficientSAM3 concept;
    • raw outputs next to the decoded result, letterbox placement and the versioned masks payload.
  • Other pages:
    • nodes/preproc: placement;
    • Model options guide: multimask_output and boxdecode_raw_outputs;
    • architecture: the BoxDecode stage members and the masks wire;
    • 0.6.1 release notes.
  • Translations: Japanese, Korean, Ukrainian and Traditional Chinese are updated, with refreshed source hashes.

Migration Impact

  • No migration impact
  • Migration required and documented

Migration notes:

  • Rebuild C++ applications, plugins and Python bindings against ABI 10. Do not load ABI 9 or earlier binaries through compatibility symlinks.
  • No source change is needed. All new options default to existing behavior.

Breaking API Change

  • Not applicable

  • This PR changes public API signatures under include/* (object layout only; no signature is renamed, removed or reordered)

  • Affected headers and symbols:

    • include/model/PreprocessPlan.h: ResizeSpec (+placement), new LetterboxPlacement;
    • include/model/Model.h: Model::Options (+boxdecode_raw_outputs, +multimask_output);
    • include/pipeline/BoxDecodeType.h: values 28–31;
    • include/pipeline/DetectionTypes.h: SegmentationMasksHeaderV1.
  • Downstream breakage risk: binaries built against ABI 9 must be rebuilt; the SONAME change makes the loader refuse a mix. There are no source-level breaks.

  • Migration steps: rebuild against libsima_neat.so.10 and the matching headers and Python bindings.

  • Planned release/version for the break: 0.6.1 (unreleased), which already carries ABI 7–9 from feat(preproc): expose fixed source arithmetic options #1337, feat(preproc): support center crop and shorter-side resize #1342 and feat(nodes): add ResizeArgMax and Model semantic segmentation decode #1353.

  • Explicit maintainer approval reference: pending review on this PR.

API / Deprecation Checklist

  • Public API unchanged
  • Public API changed with deprecation path documented in docs/how-to (additive; no deprecation needed)
  • Naming contract respected (docs/architecture/naming.md)

Release Hygiene Checklist

  • scripts/ci/check_naming_and_conflicts.sh passed
  • scripts/ci/check_artifacts.sh passed
  • scripts/ci/check_release_policy.sh passed

Limitations

  • Kernels and firmware. Runtime use needs a Kernels release with feat: sync NEAT install between eLxr SDK and paired DevKit #119's profiles. The EV74 firmware cohort check needs firmware that matches feat: sync NEAT install between eLxr SDK and paired DevKit #119; decoding itself runs on the A65.
  • SAM upstream postprocessing. Low-resolution hole and sprinkle cleanup (sam_hole_max_area / sam_sprinkle_max_area) is not a Core option; Core sends 0.
  • One prompt per run. Each prompt-decoder run is one prompt.
  • One image size per runner. A runner serves one original image size.
  • EfficientSAM3 concept. The concept image model takes packed text as well as the image, so it runs on the tensor route. The application supplies the stretch and the semantic.preprocess facts; Neat's image route is built only in front of a single-input MPK.
  • MobileSAM padding. pad_value=116 approximates SAM's zero padding after ImageNet normalization; masks never cover the padding.

Merge readiness

Open as a draft. Mark it ready once both of these hold:

  • sima-neat/kernels#119 is merged and its Runner/Scheduler packages are published (no Core manifest change is needed);
  • the DevKit results in the follow-up comment pass.

Until then, the default run of unit_boxdecode_kernel_recipe_test cannot pass in Core CI.

🤖 Generated with Claude Code

manuel-roldan and others added 3 commits November 14, 2026 20:21
Add ResizeSpec::placement (LetterboxPlacement::Center by default, TopLeft)
so a Model can letterbox with all padding on the right and bottom, as
Segment Anything expects. The model route maps it to the Preproc
padding_type the frame adapter already implements; the adapter publishes
the placed rectangle and its model-to-original affine.

- InputPlanner rejects a non-Center placement without an enabled
  Letterbox resize and rejects out-of-range values; Resize transforms
  carry the placement like the rest of their spec.
- Graph save/load and the stage cache identity carry a non-default
  placement; Center keeps the existing serialized form and identity.
- Python binds pyneat.LetterboxPlacement and ResizeSpec.placement.
- BoxDecode detects a non-centered letterbox from published pads (or the
  upstream Preproc policy before the first frame) and switches detection
  profiles to the explicit model-to-original transform, seeding a cold
  graph from the same checked Preproc policy as Crop does. Profiles that
  cannot take a transform (pose, segmentation, RCNN proposals) fail
  instead of returning shifted boxes.

ResizeSpec grows a field, so the public object layout changes: the ABI
moves from 9 to 10 (deps/manifest.json abi-version; dependency pins are
unchanged), as every layout change on develop has done since 0.6.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raw outputs

One BoxDecodeType per Segment Anything model, decoded by the Kernels
sam_mask and sam_concept profiles into a versioned masks payload:

- MobileSam = 28 ("mobilesam"): prompt decoder, top-left letterbox.
- EdgeTam = 29 ("edgetam"): mask decoder, stretch; the object-score gate
  is on when the decoder exports that output.
- EfficientSam = 30 ("efficientsam"): three multimask tokens, stretch.
- EfficientSam3 = 31 ("efficientsam3"): prompt decoder, or concept image
  model when its outputs are detections [Q,6] plus Q-channel logits;
  stretch.

Model::Options::multimask_output (default true; Python
ModelOptions.multimask_output) mirrors upstream predict(multimask_output):
true keeps the best predicted IoU of the multimask channels (1..3 of four,
0..2 of three); false keeps mask 0 (MobileSAM) or applies the SAM2
stability rule (EdgeTAM, four-mask EfficientSAM3). EfficientSAM, a
three-mask EfficientSAM3 decoder and concept detection reject false; a
one-channel export uses its mask; non-SAM decode types reject false.

Heads are bound by position and shape. Geometry comes from the decoder
inputs' semantic.preprocess; a mismatch names the encoder preprocess
setting the model needs. Masks are written at the original image size
after a SegmentationMasksHeaderV1 prefix; the headerless 160x160 payload
and YOLO-seg requests are byte-identical.

Model::Options::boxdecode_raw_outputs also returns the model's raw
outputs after the BoxDecode tensor, as the route without BoxDecode
returns them, from the same inference run (EdgeTAM tracking).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tputs

Per-model table (decode type, encoder Model preprocess, decoder options),
the multimask_output mapping and its rejections, what each output means,
and Python examples for MobileSAM, EdgeTAM tracking with raw outputs,
EfficientSAM and EfficientSAM3 concept detection. Adds the
multimask_output and boxdecode_raw_outputs rows to the Model options
guide and the BoxDecode stage note to the architecture page.

The 0.6.1 release notes describe the Segment Anything decode types and
top-left placement and state ABI 10; the pages that name the current ABI
(graph, Preproc, BoxDecode decode types) now say 10. Japanese, Korean,
Ukrainian and Traditional Chinese translations and source hashes updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
manuel-roldan and others added 7 commits November 14, 2026 22:44
…arrier

The EfficientSAM3 image model's detections leave the MLA as BF16
[1,200,8] (six used values and two padding per query) and are sliced to
[1,200,6]. AFE names that slice
"slice_MLA_0_unpack/tuple_get_item_0_slice_transform", and the lineage
classified stages by substrings of kernel|name|id, so the "unpack" in the
name made the slice an unpack stage whose output is the carrier. The
head was described as the sliced [1,200,6] view (2400 B) instead of the
[1,200,8] carrier (3200 B), and BoxDecode read 12-byte rows over 16-byte
rows: column 4 was each query's score only for every fourth query.

Classify by the kernel token when it names slice or unpack; the plugin
name decides only otherwise. The detections port is now [1,200,6] with
strides [3200,16,2], the form of the MobileSAM and EdgeTAM ports.

The extractor family test asserts each SAM head's physical carrier and
compiled pitch, and the recipe test the strided detections port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With boxdecode_raw_outputs, each raw member's checked kernel output had
the HWC view's shape ([256,256,4]) while the stage publishes the tensor
as the route without BoxDecode returns it ([1,256,256,4]), so graph build
failed with "publication shape differs from checked kernel output" for
every model.

The member's output spec is now the published shape. A cast or dequant
reads the same view with that shape (unit axes added or dropped, strides
unchanged); a same-dtype copy keeps reading the HWC view. A raw output
that is not the input with unit axes added or removed is rejected.

The recipe test checks every raw member against the stage publication
Core builds (the Scheduler's identity rule: exact shape and dense strides).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The board's dmabuf-plan admission rejected the decoder fixtures in
unit_model_output_spec_contract_test ([missing-mla-executable] MLA_0):
they named an MLA executable but shipped only share/placeholder.elf.
write_mpk_topology_elfs writes topology evidence for every MLA executable
an MPK names, with the compiler's port sections (data.ifm.b0 or
data.ifm.persistent.input_N/..., likewise for outputs) and node sizes;
write_topology_elf keeps its output byte for byte.

The EdgeTAM fixture now has the plugin graph of the CI-built decoder:
one packed MLA output, unpack_transform, slices and casts.

test_sam_concept_keeps_several_masks compares the float32 scores with
float32 values instead of float64 literals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Segment Anything BoxDecode runner publishes masks of one original image
size, but Model::run reused its first runner for every call, so a 1080p
frame after a 720p one failed with "a new frame size requires a new
runner", although the docs promise a rebuild.

Model::run now keys its runner by the original size the inputs publish
when the route ends in a Segment Anything BoxDecode, and builds another
runner when it changes. Other routes keep one runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogram .elf

unit_model_output_spec_contract_test failed on the board with "failed to
create synthetic MLA ELF": its decoder fixtures are archived without the
placeholder ELF, so share/ did not exist when write_mpk_topology_elfs
wrote the executable an MPK names. The writer now creates the
executable's directory and reports the path when it still cannot write.

The EfficientSAM fixture's MLA executable was "kernel_name_tbd", which the
ModelPack loader rejects as a package without an .elf; it is now
efficientsam_prompt_decoder_stage1_mla.elf, as the EdgeTAM fixture's is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… them

The EfficientSAM3 image model has two MLA outputs and no unpack plugin:
detections in MLA_0_0 (sliced by "slice_MLA_0_unpack/
tuple_get_item_0_slice_transform") and logits in MLA_0_1. The MLA stage
publishes both in one allocation, logits at offset 0 and detections at
14745600. The detections head took the slice plugin's name as its source
segment, a name no stage publishes in the BoxDecode route, so its binding
stayed unresolved and the frame layout fell back to memory 0 at offset 0:
BoxDecode (and the raw detections output) read the first 3200 bytes of
the logits, no query scored above any threshold, and the count was 0.

With several MLA outputs and no unpack boundary, a head now binds the MLA
output segment that holds it, as single-output models already do.

Tests: the extractor family test requires every SAM head to bind a
segment an MLA output publishes, and the render test renders the
EfficientSAM3 concept route and requires each BoxDecode input to read its
MLA output's frame span (both fail on the previous code).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…blishes

boxdecode_raw_outputs named each raw tensor with its graph endpoint token
("cast_7_detections", "dequantize_94_prompt_decoder_masks"), while the
same Model without BoxDecode publishes "cast_7/detections" and
"dequantize_94/prompt_decoder_masks". The raw outputs now take the
published tensor names; Model graph endpoints keep their tokens.

unit_model_output_spec_contract_test checks the EdgeTAM raw names
exactly (fails on the previous code with 'cast_5_mask_logits').

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@manuel-roldan

Copy link
Copy Markdown
Contributor Author

Board results: Modalix DevKit, head fe24a13

All checks pass on the device at fe24a13. The board was then restored to its original packages and stock firmware, and the restore was verified.

Distribution under test

  • Core fe24a13 (this PR).
  • sima-neat/kernels#119 at 10f09ea, with its composed EV74 firmware.
  • Runner, Runtime, Scheduler and Codecs built against that pair.

Restore check: the full dpkg list and the sha256 of every installed package file (1,134 entries) match the pre-install snapshot. The firmware hash, remoteproc and devfreq state match too, and a stock MobileSAM smoke run passes.

What the earlier device runs found

Two earlier runs exposed Core defects. Each is fixed by a commit appended to this branch, with a test that fails on the old code.

Defect seen on the device Fix
EfficientSAM3 concept returned 0 detections. The detections input was bound by the name of the slice plugin that BoxDecode replaces; no stage publishes that name in this route, so it fell back to offset 0 of the shared MLA buffer, where the logits are. The row pitch of the padded [1,200,8] BF16 carrier was also wrong. 3d646e8, 721555d
boxdecode_raw_outputs failed at graph build: kernel members were [256,256,4], but the publication expected [1,256,256,4]. c4bd4d4
Model::run reused its runner when a SAM frame size changed. 40721d9
Raw outputs used graph endpoint tokens (cast_7_detections), not the published names (cast_7/detections). fe24a13
Test fixtures: the EdgeTAM fixture had a placeholder ELF; the topology-ELF writer did not create its directory; one Python check compared floats loosely. 84e15af, 4ca9197

Results at fe24a13

Core aarch64 unit tests (27 binaries, incl. kernel recipe, static-contract extractor,
  render manifest, output spec, geometry, publication, payload decoding)   27/27 PASS
Python tests                                                                89 passed, 24 skipped (real-model fixtures not installed)

MobileSAM (INT8 decoder, top-left letterbox)
  720p and 1080p, multimask_output true/false, one Model across 720p -> 1080p -> 720p   PASS
  masks vs exact INT8-code composition                                        0 mismatches in all 5 cases
  raw outputs (both multimask values): names, dtypes, shapes, bytes          equal to the route without BoxDecode
  rejections: decoder input without preprocess metadata; built runner given another frame size   PASS
  throughput, EV74 preprocess + MLA + BoxDecode                               20.76 FPS, p50 62.7 ms (MLA-bound)

EdgeTAM (FP32, stretch)
  one click (multimask true/false), two points (stability rule), zero features,
  object-score gate with no prompt (score -0.95 -> empty mask)               6 cases, 0 mismatches
  raw outputs                                                                 equal to the route without BoxDecode

EfficientSAM-T (w8a8 and w16a16, stretch)
  centre click, box prompt                                                    0 mismatches (w8a8 vs INT8 codes)
  multimask_output=false                                                      rejected by name
  raw outputs                                                                 equal to the route without BoxDecode

EfficientSAM3 concept ("boat", 200 queries)
  top_k 5,  score_threshold 0.5   1 detection (query 144, 0.652), as the reference keeps
  top_k 20, score_threshold 0     the 20 reference queries in reference order
  boxes                                                                       equal to the head boxes scaled to the frame
  masks vs exact reference                                                    0 mismatches over 21 masks
  raw detections and logits: names, dtypes, shapes, bytes                    equal to the route without BoxDecode
  multimask_output=false                                                      rejected by name

INT8 exactness. The kernel's contract is the exact sign of the composed bilinear on the integer codes. Against that reference, every INT8 case has 0 mismatches.

Against a reference built from the FP32-dequantized logits, a few pixels differ: MobileSAM 12–35 per frame, EfficientSAM w8a8 box prompt 2. Every one of these pixels has a relative margin ≤ 2e-8, the rounding of the dequantized FP32 values, and none is an exact tie. The BF16 and FP32 models show no such differences.

Merge readiness

No open items remain on the Core side. This PR stays in draft until sima-neat/kernels#119 is merged and published, so Core CI resolves the sam_mask and sam_concept profiles.

🤖 Generated with Claude Code

@manuel-roldan
manuel-roldan marked this pull request as ready for review November 15, 2026 16:53
@manuel-roldan
manuel-roldan requested a review from a team as a code owner November 15, 2026 16:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T00:21:54.799593Z e4bb712 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe24a13a3c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +294 to +298
const auto& geometry = std::get<Geometry>(parsed);
if ((geometry.width || geometry.height) &&
(geometry.width != original_width || geometry.height != original_height))
return invalid("the prepared output is sized for another original image; a new "
"frame size requires a new runner");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate preprocessing geometry on every SAM invocation

When a prepared SAM runner receives tensors for the same original image dimensions but with different preprocessing, this guard still returns the original executable. For example, after preparing MobileSAM with top-left letterboxing, a subsequent frame with centered letterboxing passes because only width and height are compared, although sam_boxdecode_geometry() would reject it during cold admission. The kernel then interprets the centered logits using its fixed top-left geometry and silently publishes incorrectly positioned masks. Validate the current frame's resize mode, canvas and placement against the admitted SAM geometry before returning the prepared selection.

Useful? React with 👍 / 👎.

manuel-roldan and others added 3 commits November 15, 2026 21:46
…as SAM does

A Model-level LetterboxPlacement::TopLeft placed a 640x427 frame on the 1024
canvas as 1024x684: the frame adapter's contain rule rounds the non-limiting
axis up in FP32. Segment Anything, and the MobileSAM BoxDecode crop, use
int(side * canvas / longest + 0.5) in double, which gives 683 rows and 341
padding rows.

The Model now marks its top-left letterbox in the compiled Preproc contract
(internal ProcessCvuStagePayload::letterbox_nearest_extent). The frame adapter
then fills the canvas with the limiting axis and rounds the other to the
nearest pixel, with the kernel's arithmetic (internal::nearest_letterbox_extent).
The placed rectangle, the pads and the per-axis affine that semantic.preprocess
publishes follow from that rectangle.

Centered letterboxes, and every placement the Preproc node takes through
padding_type, keep the legacy upward rounding, so YOLO and other existing
routes are unchanged. Model::benchmark's synthetic MobileSAM geometry uses the
same helper and publishes the same per-axis affine.

Tests (both fail on the previous code):
- unit_preproc_contract_rules_test: Model-route facts on a 1024 canvas for
  640x427, 427x640, 1280x720, 1001x667, 667x1001 and 1920x1081 top-left, and
  640x427, 427x640 and 1001x667 centered (684 and 683 stay rounded up).
- unit_benchmark_measurement_contract_test: 640x427 and 427x640 MobileSAM
  benchmark geometry.

Docs: the preproc placement paragraph states the rounding, in English and the
four translations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… is off

The EdgeTAM tracking section told readers to take object_pointers[0, class]
for the memory bank. That holds only for multimask_output=True. With
multimask_output=False, SAM 2 (and the EdgeTAM README and calibration) keeps
the single-mask pointer 0, even when the stability fallback chooses a mask from
planes 1-3. The text and the example now say so, in English and the four
translations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Segment Anything prompt decoder takes the image geometry from
semantic.preprocess on its inputs, so Model::build() and execution_plan()
without inputs cannot prepare it. The error now says that its Model builds
only from inputs and names run(), build(inputs) and execution_plan(inputs).
The BoxDecode docs say the same, in English and the four translations.

A no-input plan cannot describe the route truthfully: the decoder's output
size is the original image size, which only the inputs give.

Test: unit_boxdecode_kernel_recipe_test (sam_requires_input_geometry) fails
on the previous message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@manuel-roldan

manuel-roldan commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Round 4: e4bb712

Three commits, found while moving the models repository's Segment Anything scripts to these decode types:

  • 22bf9c1. A Model top-left letterbox now rounds the shorter side as SAM does: 640×427 becomes 1024×683, not 684. It matches the MobileSam crop, so masks on such frames are SAM-exact.
    • This affected about 29% of COCO val images; 1280×720 was never affected.
    • Centered letterboxes keep rounding up.
    • Tests in unit_preproc_contract_rules_test and unit_benchmark_measurement_contract_test fail on the old code.
  • c5bb535. The EdgeTAM tracking docs now say to keep object pointer 0 when multimask_output is false, as SAM 2 does.
  • e4bb712. A SAM decoder Model built without inputs now fails with a message that says to pass inputs to run(), build(inputs) or execution_plan(inputs).

Board check (Modalix DevKit, e4bb712 with kernels#119), MobileSAM w16a16 on COCO val2017 box prompts (5000 images, 36,335 prompts):

Pipeline Mask AP Throughput
CPU pre and post (Pillow, OpenCV) 55.20 9.53 FPS / 186.5 ms
Neat Preproc (SAM-exact) + MobileSam BoxDecode 55.17 13.90 FPS / 96.9 ms
  • The −0.03 AP gap comes from the different preprocessing: Pillow uint8 bilinear vs SAM-exact antialiased. It is within the metric's sampling noise (jackknife SE 0.06–0.18 AP); board runs themselves are deterministic.
  • BoxDecode and the CPU post give the same AP from identical decoder outputs.
  • The board was restored and verified afterwards.

🤖 Generated with Claude Code

This branch was successfully deployed

2 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant