Skip to content

feat(examples): add EdgeTAM video object segmenter - #726

Draft
Onat-Inak wants to merge 1 commit into
developfrom
feat/edgetam-video-object-segmenter
Draft

Onat-Inak wants to merge 1 commit into
developfrom
feat/edgetam-video-object-segmenter

Conversation

@Onat-Inak

@Onat-Inak Onat-Inak commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Why

Models ships EdgeTAM as four packages (sima-neat/models#823), and Apps has no example that uses them. No example follows one object through a video, and none runs a model that carries state from frame to frame. EdgeTAM also needs a prompt on the first frame, which normally comes from a click, so a developer without a UI has nothing to start from. This example reads the prompt as two numbers from config.yaml.

Closes #725

Usage

Download the model, then run. The packaged config already points at the four packages and at a shipped sample video, so it runs with no edit:

sima-cli models download --id edgetam --variant w16a16_w8a16_w16a16_w8a8 -o models
unzip models/edgetam/w16a16_w8a16_w16a16_w8a8/edgetam_w16a16_w8a16_w16a16_w8a8_default.zip -d models/edgetam/w16a16_w8a16_w16a16_w8a8
./examples/segmentation/edgetam-video-object-segmenter/src/cpp/pre-built/edgetam-video-object-segmenter --config examples/segmentation/edgetam-video-object-segmenter/src/common/config.yaml
python3 examples/segmentation/edgetam-video-object-segmenter/src/python/main.py --config examples/segmentation/edgetam-video-object-segmenter/src/common/config.yaml

To follow another object, set the point on it in the first frame:

source:
  video_path: assets/datasets/tum-rgbd/freiburg1-desk.mp4
prompt:
  x: 0.87        # from the left edge, as a fraction of the frame width
  y: 0.38        # from the top edge, as a fraction of the frame height
inference:
  frames: 150    # 0 means the whole video
output:
  dir: sandbox/edgetam-video-object-segmenter
  save_every: 10 # 0 saves no images

The application writes results.json with each frame's visibility, mask area and mask centre, and every tenth frame with the mask drawn on it. Frames 0, 20 and 40 of the default run, with the ring on the prompt point in the first:

EdgeTAM video object segmenter overlays

What Changed

  • New example examples/segmentation/edgetam-video-object-segmenter in C++ and Python, with one main per language and the same functions, constants, messages and outputs in both.
  • main loads the four models, builds one runner for each, and runs them in order for every frame. The Neat calls are all in main; utils/tracker holds only the CPU steps: the prompt encoding, the mask choice, the memory mask and the memory bank.
  • The image encoder uses Neat preprocessing on the RGB frame: a bicubic stretch to 1024x1024 and ImageNet normalization, as the model's README specifies.
  • When the mask decoder's object score says the object is absent, the frame is reported with visible: false and the tracker keeps looking in the next frame.
  • The test scope takes the four packages from the Model Registry's main branch.
video file --> image encoder --> memory attention --> mask decoder --> mask --> results.json, overlay images
                                        ^                  ^             |
                                   memory bank        prompt point       v
                                        ^            (first frame)  memory encoder
                                        |                                |
                                        +--------------------------------+

The four models are separate runners and not one graph because each frame's memory attention reads the memory the previous frame produced, and the CPU chooses the mask between the mask decoder and the memory encoder.

Validation

Modalix DevKit, Platform 3.0.0, Neat 0.6.1. The packages are the w16a16_w8a16_w16a16_w8a8 build that Models CI produced for sima-neat/models#823.

  • SDK build: the example and its tests build with 0 warnings in the Neat SDK.
  • DevKit, unit tests: Python 16 passed; the C++ unit test passed with 30 checks. Both cover the prompt encoding, the mask choice (best of masks 1 to 3, no object), the memory mask, the object location, the memory bank order and limits, and the config errors.
  • DevKit, e2e tests: Python 2 passed; C++ passed with 11 checks. On the shipped video the blue book must be visible in frames 0 to 40 with its centre within 0.06 of measured positions in frames 0, 20 and 40, no object may be reported in frames 60 to 99 after the book leaves the view, and every tenth frame must be saved. A second run with another prompt point must cover the monitor instead. E2e ran with the test binaries and pytest directly, because the scope keeps it disabled.
  • Against the model's reference tracker: with the same preprocessing options, the Python application matched Models' performance.py tracker on all 150 frames: the same visibility, area and centre to four decimals. With bicubic resizing, visibility differs in 2 of 150 frames and the area by at most 0.0016 of the frame.
  • Languages agree: on the default run both report the object visible in 61 of 150 frames. Visibility differs in 2 frames, the area by at most 0.0047 of the frame and the centre by at most 0.0044. The two decode the video with different OpenCV versions (4.10 and 5.0), whose first frames differ by one or two levels per pixel.
  • Host: clang-format, check_cmake_style.py, check_duplicate_includes.py, check_conflicts.sh, scripts/ci/check_public_includes.sh, scripts/validate_readmes.py (32 READMEs) and scripts/generate_catalog.py pass.

Not verified:

  • The README's download command as written. EdgeTAM is merged in Models but its package is not yet promoted, so sima-cli models download cannot fetch it. The zip name in the README follows the registry's naming and was not observed.
  • E2e in CI. Both suites are disabled in tests/test-scope.yaml with that reason and need enabled: true once the package is on main.
  • sima-cli neat install apps@feat/edgetam-video-object-segmenter and the installed README flow. The runs above used the source tree and the SDK build tree.
  • Vulcan CI for this branch had not run when this description was written.
  • Speed at the benchmark clock. The runs tracked about 5 frames per second with the MLA clock at 600 MHz; nothing was measured at 1 GHz.
  • The portal's catalog and detail views of the preview image.

Skill updates

Skill updates: none. The example uses Model, build and run as references/model-graph-run.md already describes, and for tensor inputs sets only the input kind.

No example followed one object through a video, and EdgeTAM needs a
first-frame prompt that normally comes from a click. This offline
example reads the prompt point from config.yaml, so it needs no UI.

- Runs EdgeTAM's four models per frame with the Neat Model API and
  keeps the memory bank on the CPU, in C++ and Python.
- Saves mask overlays and a results.json with each frame's
  visibility, mask area and mask centre.
- E2e stays disabled until the EdgeTAM package is promoted to the
  main branch of the Model Registry.

Closes #725
@Onat-Inak Onat-Inak self-assigned this Nov 14, 2026

This branch was successfully deployed

3 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant