Repository navigation
feat(boxdecode): Segment Anything masks for MobileSAM, EdgeTAM, EfficientSAM and EfficientSAM3 - #1355
manuel-roldan wants to merge 13 commits into
Conversation
Add ResizeSpec::placement (LetterboxPlacement::Center by default, TopLeft) so a Model can letterbox with all padding on the right and bottom, as Segment Anything expects. The model route maps it to the Preproc padding_type the frame adapter already implements; the adapter publishes the placed rectangle and its model-to-original affine. - InputPlanner rejects a non-Center placement without an enabled Letterbox resize and rejects out-of-range values; Resize transforms carry the placement like the rest of their spec. - Graph save/load and the stage cache identity carry a non-default placement; Center keeps the existing serialized form and identity. - Python binds pyneat.LetterboxPlacement and ResizeSpec.placement. - BoxDecode detects a non-centered letterbox from published pads (or the upstream Preproc policy before the first frame) and switches detection profiles to the explicit model-to-original transform, seeding a cold graph from the same checked Preproc policy as Crop does. Profiles that cannot take a transform (pose, segmentation, RCNN proposals) fail instead of returning shifted boxes. ResizeSpec grows a field, so the public object layout changes: the ABI moves from 9 to 10 (deps/manifest.json abi-version; dependency pins are unchanged), as every layout change on develop has done since 0.6.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raw outputs
One BoxDecodeType per Segment Anything model, decoded by the Kernels
sam_mask and sam_concept profiles into a versioned masks payload:
- MobileSam = 28 ("mobilesam"): prompt decoder, top-left letterbox.
- EdgeTam = 29 ("edgetam"): mask decoder, stretch; the object-score gate
is on when the decoder exports that output.
- EfficientSam = 30 ("efficientsam"): three multimask tokens, stretch.
- EfficientSam3 = 31 ("efficientsam3"): prompt decoder, or concept image
model when its outputs are detections [Q,6] plus Q-channel logits;
stretch.
Model::Options::multimask_output (default true; Python
ModelOptions.multimask_output) mirrors upstream predict(multimask_output):
true keeps the best predicted IoU of the multimask channels (1..3 of four,
0..2 of three); false keeps mask 0 (MobileSAM) or applies the SAM2
stability rule (EdgeTAM, four-mask EfficientSAM3). EfficientSAM, a
three-mask EfficientSAM3 decoder and concept detection reject false; a
one-channel export uses its mask; non-SAM decode types reject false.
Heads are bound by position and shape. Geometry comes from the decoder
inputs' semantic.preprocess; a mismatch names the encoder preprocess
setting the model needs. Masks are written at the original image size
after a SegmentationMasksHeaderV1 prefix; the headerless 160x160 payload
and YOLO-seg requests are byte-identical.
Model::Options::boxdecode_raw_outputs also returns the model's raw
outputs after the BoxDecode tensor, as the route without BoxDecode
returns them, from the same inference run (EdgeTAM tracking).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tputs Per-model table (decode type, encoder Model preprocess, decoder options), the multimask_output mapping and its rejections, what each output means, and Python examples for MobileSAM, EdgeTAM tracking with raw outputs, EfficientSAM and EfficientSAM3 concept detection. Adds the multimask_output and boxdecode_raw_outputs rows to the Model options guide and the BoxDecode stage note to the architecture page. The 0.6.1 release notes describe the Segment Anything decode types and top-left placement and state ABI 10; the pages that name the current ABI (graph, Preproc, BoxDecode decode types) now say 10. Japanese, Korean, Ukrainian and Traditional Chinese translations and source hashes updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arrier The EfficientSAM3 image model's detections leave the MLA as BF16 [1,200,8] (six used values and two padding per query) and are sliced to [1,200,6]. AFE names that slice "slice_MLA_0_unpack/tuple_get_item_0_slice_transform", and the lineage classified stages by substrings of kernel|name|id, so the "unpack" in the name made the slice an unpack stage whose output is the carrier. The head was described as the sliced [1,200,6] view (2400 B) instead of the [1,200,8] carrier (3200 B), and BoxDecode read 12-byte rows over 16-byte rows: column 4 was each query's score only for every fourth query. Classify by the kernel token when it names slice or unpack; the plugin name decides only otherwise. The detections port is now [1,200,6] with strides [3200,16,2], the form of the MobileSAM and EdgeTAM ports. The extractor family test asserts each SAM head's physical carrier and compiled pitch, and the recipe test the strided detections port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With boxdecode_raw_outputs, each raw member's checked kernel output had the HWC view's shape ([256,256,4]) while the stage publishes the tensor as the route without BoxDecode returns it ([1,256,256,4]), so graph build failed with "publication shape differs from checked kernel output" for every model. The member's output spec is now the published shape. A cast or dequant reads the same view with that shape (unit axes added or dropped, strides unchanged); a same-dtype copy keeps reading the HWC view. A raw output that is not the input with unit axes added or removed is rejected. The recipe test checks every raw member against the stage publication Core builds (the Scheduler's identity rule: exact shape and dense strides). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The board's dmabuf-plan admission rejected the decoder fixtures in unit_model_output_spec_contract_test ([missing-mla-executable] MLA_0): they named an MLA executable but shipped only share/placeholder.elf. write_mpk_topology_elfs writes topology evidence for every MLA executable an MPK names, with the compiler's port sections (data.ifm.b0 or data.ifm.persistent.input_N/..., likewise for outputs) and node sizes; write_topology_elf keeps its output byte for byte. The EdgeTAM fixture now has the plugin graph of the CI-built decoder: one packed MLA output, unpack_transform, slices and casts. test_sam_concept_keeps_several_masks compares the float32 scores with float32 values instead of float64 literals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Segment Anything BoxDecode runner publishes masks of one original image size, but Model::run reused its first runner for every call, so a 1080p frame after a 720p one failed with "a new frame size requires a new runner", although the docs promise a rebuild. Model::run now keys its runner by the original size the inputs publish when the route ends in a Segment Anything BoxDecode, and builds another runner when it changes. Other routes keep one runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogram .elf unit_model_output_spec_contract_test failed on the board with "failed to create synthetic MLA ELF": its decoder fixtures are archived without the placeholder ELF, so share/ did not exist when write_mpk_topology_elfs wrote the executable an MPK names. The writer now creates the executable's directory and reports the path when it still cannot write. The EfficientSAM fixture's MLA executable was "kernel_name_tbd", which the ModelPack loader rejects as a package without an .elf; it is now efficientsam_prompt_decoder_stage1_mla.elf, as the EdgeTAM fixture's is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… them The EfficientSAM3 image model has two MLA outputs and no unpack plugin: detections in MLA_0_0 (sliced by "slice_MLA_0_unpack/ tuple_get_item_0_slice_transform") and logits in MLA_0_1. The MLA stage publishes both in one allocation, logits at offset 0 and detections at 14745600. The detections head took the slice plugin's name as its source segment, a name no stage publishes in the BoxDecode route, so its binding stayed unresolved and the frame layout fell back to memory 0 at offset 0: BoxDecode (and the raw detections output) read the first 3200 bytes of the logits, no query scored above any threshold, and the count was 0. With several MLA outputs and no unpack boundary, a head now binds the MLA output segment that holds it, as single-output models already do. Tests: the extractor family test requires every SAM head to bind a segment an MLA output publishes, and the render test renders the EfficientSAM3 concept route and requires each BoxDecode input to read its MLA output's frame span (both fail on the previous code). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…blishes
boxdecode_raw_outputs named each raw tensor with its graph endpoint token
("cast_7_detections", "dequantize_94_prompt_decoder_masks"), while the
same Model without BoxDecode publishes "cast_7/detections" and
"dequantize_94/prompt_decoder_masks". The raw outputs now take the
published tensor names; Model graph endpoints keep their tokens.
unit_model_output_spec_contract_test checks the EdgeTAM raw names
exactly (fails on the previous code with 'cast_5_mask_logits').
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Board results: Modalix DevKit, head fe24a13All checks pass on the device at fe24a13. The board was then restored to its original packages and stock firmware, and the restore was verified. Distribution under test
Restore check: the full dpkg list and the sha256 of every installed package file (1,134 entries) match the pre-install snapshot. The firmware hash, remoteproc and devfreq state match too, and a stock MobileSAM smoke run passes. What the earlier device runs foundTwo earlier runs exposed Core defects. Each is fixed by a commit appended to this branch, with a test that fails on the old code.
Results at fe24a13INT8 exactness. The kernel's contract is the exact sign of the composed bilinear on the integer codes. Against that reference, every INT8 case has 0 mismatches. Against a reference built from the FP32-dequantized logits, a few pixels differ: MobileSAM 12–35 per frame, EfficientSAM w8a8 box prompt 2. Every one of these pixels has a relative margin ≤ 2e-8, the rounding of the dequantized FP32 values, and none is an exact tie. The BF16 and FP32 models show no such differences. Merge readinessNo open items remain on the Core side. This PR stays in draft until sima-neat/kernels#119 is merged and published, so Core CI resolves the 🤖 Generated with Claude Code |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe24a13a3c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const auto& geometry = std::get<Geometry>(parsed); | ||
| if ((geometry.width || geometry.height) && | ||
| (geometry.width != original_width || geometry.height != original_height)) | ||
| return invalid("the prepared output is sized for another original image; a new " | ||
| "frame size requires a new runner"); |
There was a problem hiding this comment.
Validate preprocessing geometry on every SAM invocation
When a prepared SAM runner receives tensors for the same original image dimensions but with different preprocessing, this guard still returns the original executable. For example, after preparing MobileSAM with top-left letterboxing, a subsequent frame with centered letterboxing passes because only width and height are compared, although sam_boxdecode_geometry() would reject it during cold admission. The kernel then interprets the centered logits using its fixed top-left geometry and silently publishes incorrectly positioned masks. Validate the current frame's resize mode, canvas and placement against the admitted SAM geometry before returning the prepared selection.
Useful? React with 👍 / 👎.
…as SAM does A Model-level LetterboxPlacement::TopLeft placed a 640x427 frame on the 1024 canvas as 1024x684: the frame adapter's contain rule rounds the non-limiting axis up in FP32. Segment Anything, and the MobileSAM BoxDecode crop, use int(side * canvas / longest + 0.5) in double, which gives 683 rows and 341 padding rows. The Model now marks its top-left letterbox in the compiled Preproc contract (internal ProcessCvuStagePayload::letterbox_nearest_extent). The frame adapter then fills the canvas with the limiting axis and rounds the other to the nearest pixel, with the kernel's arithmetic (internal::nearest_letterbox_extent). The placed rectangle, the pads and the per-axis affine that semantic.preprocess publishes follow from that rectangle. Centered letterboxes, and every placement the Preproc node takes through padding_type, keep the legacy upward rounding, so YOLO and other existing routes are unchanged. Model::benchmark's synthetic MobileSAM geometry uses the same helper and publishes the same per-axis affine. Tests (both fail on the previous code): - unit_preproc_contract_rules_test: Model-route facts on a 1024 canvas for 640x427, 427x640, 1280x720, 1001x667, 667x1001 and 1920x1081 top-left, and 640x427, 427x640 and 1001x667 centered (684 and 683 stay rounded up). - unit_benchmark_measurement_contract_test: 640x427 and 427x640 MobileSAM benchmark geometry. Docs: the preproc placement paragraph states the rounding, in English and the four translations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… is off The EdgeTAM tracking section told readers to take object_pointers[0, class] for the memory bank. That holds only for multimask_output=True. With multimask_output=False, SAM 2 (and the EdgeTAM README and calibration) keeps the single-mask pointer 0, even when the stability fallback chooses a mask from planes 1-3. The text and the example now say so, in English and the four translations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Segment Anything prompt decoder takes the image geometry from semantic.preprocess on its inputs, so Model::build() and execution_plan() without inputs cannot prepare it. The error now says that its Model builds only from inputs and names run(), build(inputs) and execution_plan(inputs). The BoxDecode docs say the same, in English and the four translations. A no-input plan cannot describe the route truthfully: the decoder's output size is the original image size, which only the inputs give. Test: unit_boxdecode_kernel_recipe_test (sam_requires_input_geometry) fails on the previous message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 4: e4bb712Three commits, found while moving the models repository's Segment Anything scripts to these decode types:
Board check (Modalix DevKit, e4bb712 with kernels#119), MobileSAM w16a16 on COCO val2017 box prompts (5000 images, 36,335 prompts):
🤖 Generated with Claude Code |
Refs sima-neat/models#390
Summary
This PR decodes Segment Anything models through BoxDecode. There is one decode type per model. A run returns the chosen mask at the original image size, with its box and predicted IoU, in a versioned masks payload.
decode_segmentationreads that payload.MobileSAM (sima-neat/models#390) is the issue this PR serves. The other model PRs it was checked against are:
Depends on sima-neat/kernels#119. That PR adds the Kernels BoxDecode profiles
sam_maskandsam_concept. Core builds the named request; Kernels admits it, selects the mask, resamples it and applies the sign test. See Dependency and CI below.The PR has three commits, meant to be reviewed in order:
feat(preproc): top-left letterbox placement, and the ABI bump to 10.feat(boxdecode): the Segment Anything decode types,multimask_outputandboxdecode_raw_outputs, with tests.docs(boxdecode): reference docs, release notes and the four translations.Decode types
BoxDecodeType(token)MobileSam= 28 (mobilesam)resize.mode=Letterbox,resize.placement=TopLeft, 1024×1024,pad_value=116,preset=ImageNetmultimask_outputEdgeTam= 29 (edgetam)resize.mode=Stretch, 1024×1024,preset=ImageNetmultimask_output;boxdecode_raw_outputsfor trackingEfficientSam= 30 (efficientsam)resize.mode=Stretch, 1024×1024, values 0–1 only (the encoder normalizes itself)multimask_outputtrueEfficientSam3= 31 (efficientsam3)resize.mode=Stretch, 1008×1008, mean and stddev 0.5multimask_outputEfficientSam3= 31semantic.preprocesstop_k,score_thresholdmobile-sam,edge-tam,efficient-sam,efficient-sam3.pyneat.BoxDecodeType.MobileSam,EdgeTam,EfficientSam,EfficientSam3.Pseudocode for one prompt-decoder run:
The concept image model is different. It binds detections
[Q,6]and logits[h,w,Q]. It keeps up totop_kqueries abovescore_threshold, in descending score order with ties broken by query index. It returns each kept query's box from the detection head and that query's own mask.Model::Options::multimask_output(PythonModelOptions.multimask_output, defaulttrue)This option mirrors upstream
predict(multimask_output=...).decode_typetruefalseMobileSam(C=4)EdgeTam(C=4)EfficientSam(C=3)EfficientSam3, C=4 / C=3EfficientSam3concepttop_kqueriesfalsewhen the BoxDecode stage is built, aspose_classesalready does.MobileSam.Model::Options::boxdecode_raw_outputs(defaultfalse)When this is
true, the route returns the BoxDecode tensor first. After it come the model's own outputs, exactly as the same Model returns them without BoxDecode: the same names, dtypes and shapes, from the same inference run. EdgeTAM tracking needs this, because the memory encoder and bank take the chosen plane's low-resolution logits and the object pointer.How the raw copies are made:
The default route is unchanged.
Top-left letterbox placement
ResizeSpec::placementtakesLetterboxPlacement::Center(the default) orTopLeft. The Preproc frame adapter already implements top-left placement and publishes the placed rectangle and its affine.TopLeftunless the resize is an enabled letterbox.Geometry rules
semantic.preprocess. Pass the encoder outputs with.cvu().Model::runbuilds a new runner when the size changes; a streaming graph rejects a frame of another size.ABI
The ABI moves from 9 to 10 (
libsima_neat.so.10).What changed:
ResizeSpecgainsplacement.Model::Optionsgainsboxdecode_raw_outputsandmultimask_output.Why this is a bump and not a fold into 9:
abi-versionindeps/manifest.json"is incremented whenever a public C++ type layout ... changes incompatibly".What else this PR touches for the ABI:
deps/manifest.jsonchanges onlyabi-version. The dependency pins and policies are untouched.No public signature is renamed, removed or reordered. The rest is additive:
LetterboxPlacementenum;BoxDecodeTypevalues 28–31;SegmentationMasksHeaderV1inDetectionTypes.h;decode_segmentation/decode_bboxrecognizing the V1 payload by its magic.Against develop, YOLO-seg requests and the headerless 160×160 segmentation payload are byte-identical, and centered letterboxes keep their mapping.
Dependency and CI
The Kernels profiles live in sima-neat/kernels#119, which is not yet in the dependency set Core CI resolves.
What is independent of the dependency:
check_kernel.snap, so once feat: sync NEAT install between eLxr SDK and paired DevKit #119 is merged and its Runner/Scheduler packages are published, CI picks them up with no change todeps/manifest.json.What Core CI can exercise now:
multimask_outputmapping and every rejection;unit_boxdecode_kernel_recipe_test --sam-core-contractand--boxdecode-raw-outputs-core-contract, which fail before admission.What it cannot exercise until #119 is published:
unit_boxdecode_kernel_recipe_test. Its SAM and raw-output admission cases need thesam_maskandsam_conceptprofiles, and they fail at Kernels admission against the current set;The "Merge readiness" section below proposes the merge order.
Change Type
Risk
Risk notes:
boxdecode_raw_outputsis set.Test Evidence
Commands run and outcomes:
Device details, and the defects the earlier device runs found (fixed by the appended commits), are in the board results comment: #1355 (comment)
Docs Impact
Docs notes:
reference/boxdecode_decode_types:multimask_outputmapping and rejections;nodes/preproc:placement;multimask_outputandboxdecode_raw_outputs;Migration Impact
Migration notes:
Breaking API Change
Not applicable
This PR changes public API signatures under
include/*(object layout only; no signature is renamed, removed or reordered)Affected headers and symbols:
include/model/PreprocessPlan.h:ResizeSpec(+placement), newLetterboxPlacement;include/model/Model.h:Model::Options(+boxdecode_raw_outputs, +multimask_output);include/pipeline/BoxDecodeType.h: values 28–31;include/pipeline/DetectionTypes.h:SegmentationMasksHeaderV1.Downstream breakage risk: binaries built against ABI 9 must be rebuilt; the SONAME change makes the loader refuse a mix. There are no source-level breaks.
Migration steps: rebuild against
libsima_neat.so.10and the matching headers and Python bindings.Planned release/version for the break: 0.6.1 (unreleased), which already carries ABI 7–9 from feat(preproc): expose fixed source arithmetic options #1337, feat(preproc): support center crop and shorter-side resize #1342 and feat(nodes): add ResizeArgMax and Model semantic segmentation decode #1353.
Explicit maintainer approval reference: pending review on this PR.
API / Deprecation Checklist
docs/architecture/naming.md)Release Hygiene Checklist
scripts/ci/check_naming_and_conflicts.shpassedscripts/ci/check_artifacts.shpassedscripts/ci/check_release_policy.shpassedLimitations
sam_hole_max_area/sam_sprinkle_max_area) is not a Core option; Core sends 0.semantic.preprocessfacts; Neat's image route is built only in front of a single-input MPK.pad_value=116approximates SAM's zero padding after ImageNet normalization; masks never cover the padding.Merge readiness
Open as a draft. Mark it ready once both of these hold:
Until then, the default run of
unit_boxdecode_kernel_recipe_testcannot pass in Core CI.🤖 Generated with Claude Code