Repository navigation
Conversation
No example followed one object through a video, and EdgeTAM needs a first-frame prompt that normally comes from a click. This offline example reads the prompt point from config.yaml, so it needs no UI. - Runs EdgeTAM's four models per frame with the Neat Model API and keeps the memory bank on the CPU, in C++ and Python. - Saves mask overlays and a results.json with each frame's visibility, mask area and mask centre. - E2e stays disabled until the EdgeTAM package is promoted to the main branch of the Model Registry. Closes #725
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Models ships EdgeTAM as four packages (sima-neat/models#823), and Apps has no example that uses them. No example follows one object through a video, and none runs a model that carries state from frame to frame. EdgeTAM also needs a prompt on the first frame, which normally comes from a click, so a developer without a UI has nothing to start from. This example reads the prompt as two numbers from
config.yaml.Closes #725
Usage
Download the model, then run. The packaged config already points at the four packages and at a shipped sample video, so it runs with no edit:
To follow another object, set the point on it in the first frame:
The application writes
results.jsonwith each frame's visibility, mask area and mask centre, and every tenth frame with the mask drawn on it. Frames 0, 20 and 40 of the default run, with the ring on the prompt point in the first:What Changed
examples/segmentation/edgetam-video-object-segmenterin C++ and Python, with onemainper language and the same functions, constants, messages and outputs in both.mainloads the four models, builds one runner for each, and runs them in order for every frame. The Neat calls are all inmain;utils/trackerholds only the CPU steps: the prompt encoding, the mask choice, the memory mask and the memory bank.visible: falseand the tracker keeps looking in the next frame.mainbranch.The four models are separate runners and not one graph because each frame's memory attention reads the memory the previous frame produced, and the CPU chooses the mask between the mask decoder and the memory encoder.
Validation
Modalix DevKit, Platform 3.0.0, Neat 0.6.1. The packages are the
w16a16_w8a16_w16a16_w8a8build that Models CI produced for sima-neat/models#823.pytestdirectly, because the scope keeps it disabled.performance.pytracker on all 150 frames: the same visibility, area and centre to four decimals. With bicubic resizing, visibility differs in 2 of 150 frames and the area by at most 0.0016 of the frame.clang-format,check_cmake_style.py,check_duplicate_includes.py,check_conflicts.sh,scripts/ci/check_public_includes.sh,scripts/validate_readmes.py(32 READMEs) andscripts/generate_catalog.pypass.Not verified:
sima-cli models downloadcannot fetch it. The zip name in the README follows the registry's naming and was not observed.tests/test-scope.yamlwith that reason and needenabled: trueonce the package is onmain.sima-cli neat install apps@feat/edgetam-video-object-segmenterand the installed README flow. The runs above used the source tree and the SDK build tree.Skill updates
Skill updates: none. The example uses
Model,buildandrunasreferences/model-graph-run.mdalready describes, and for tensor inputs sets only the input kind.