Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,42 @@ jobs:
dvidia-motor-memory --help
dvidia-observe --help
dvidia-observation-studio --help
dvidia-motion --help
dvidia-training-doctor --json
- name: Test footage, visual training and loopback studio
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models -v
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models tests.test_motion tests.test_motion_benchmark -v

motion:
name: Causal image motion / Linux / Python 3.12
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
with:
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.12"
- name: Install CPU optical-flow runtime
run: |
sudo apt-get update
sudo apt-get install -y ffmpeg
python -m pip install ".[observations]"
python -m pip check
dvidia-motion --help
- name: Verify source clocks, causal rows, scoring and local replay
run: python -m unittest tests.test_motion tests.test_motion_video tests.test_motion_benchmark tests.test_motion_studio -v
- name: Run and inspect authored tracking and prediction fixtures
run: |
dvidia-motion benchmark --output runs/image-motion-ci
dvidia-motion inspect runs/image-motion-ci
- name: Preserve every attempted synthetic motion case
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: authored-image-motion
path: runs/image-motion-ci
if-no-files-found: ignore

observations:
name: Observation detectors / Linux / Python 3.12
Expand Down
30 changes: 29 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,31 @@ six public excerpts. This establishes the integration, not accuracy: labels can
be wrong, outcomes remain unknown, and there is no 3D reconstruction, automatic
action understanding, or human-video-to-robot control bridge.

## Estimate and forecast image motion

The [first image-motion alpha](docs/motion-observations.md) is available in the
candidate source checkout. Manually seed one first-frame region; causal CPU
feature tracking estimates its image-anchor speed and forecasts a path at
100, 250 and 500 ms. Lost tracks stay lost. Actual integer video timestamps,
tracking support, units, source/implementation hashes and failed attempts are
preserved. This feature trains no model and requires no detector weights.

```sh
.venv/bin/python -m pip install '.[observations]'
.venv/bin/dvidia-motion benchmark --output runs/image-motion-001
.venv/bin/dvidia-motion inspect runs/image-motion-001
.venv/bin/dvidia-motion studio runs/image-motion-001 --port 8286
```

Use the [candidate source installation instructions](docs/motion-observations.md#install-from-source),
not the frozen `observation-lab-v0.1.0` tag, for `dvidia-motion`. The authored
benchmark and read-only local replay test image tracking and prediction, not
real detector accuracy. Speeds are analysis-image pixels/second; physical speed,
moving-camera compensation, live robot latency and interaction prediction remain
unqualified. This is one component toward [the full Observation Lab v0.2](docs/observation-v0.2-plan.md).
See the [measured authored benchmark](benchmarks/motion-alpha-20261010/README.md),
including conditional errors, failed tracking and forecast tails.

## Try the pipeline

Generate an explicitly synthetic example, then train and verify its output:
Expand Down Expand Up @@ -132,4 +157,7 @@ Each command uses a fresh output directory where applicable. The model and compl

The demo presents saved candidate and open-jaw control playback from one exact simulation scene. Training runs locally; the dataset and model cards document reproducible inputs, CPU measurements and the limits of this research alpha. [Maintainer publishing tools](publishing/huggingface/README.md) stage only the already-public synthetic release and use a credential from macOS Keychain without printing or caching it.

MIT licensed. Research and measured limitations: [research.dvidia.org](https://research.dvidia.org). Contact: [hello@dvidia.org](mailto:hello@dvidia.org).
Original DVIDIA code is MIT licensed; observation detector adaptations retain
Apache 2.0 notices and upstream models retain their own licenses. Input footage
has separate rights. Research and measured limitations: [research.dvidia.org](https://research.dvidia.org).
Contact: [hello@dvidia.org](mailto:hello@dvidia.org).
137 changes: 137 additions & 0 deletions benchmarks/motion-alpha-20261010/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# DVIDIA image-motion alpha — 10 October 2026

**Six authored fixtures processed successfully; real-world accuracy and live
robot latency remain unmeasured.** This evaluates sparse tracking of a manually
seeded image region and passive constant-velocity forecasting. It evaluates
neither an object detector nor a trained interaction or robot-control model.

The [frozen protocol](protocol.json) and [aggregate results](result.json)
preserve parameters, input/reference hashes, software, attempted counts,
conditional errors, abstentions and timing scope. They contain no input footage,
model weights, private data or credentials. The generated fixtures and these
DVIDIA artifacts are MIT licensed. The existing project notices continue to
apply to upstream dependencies.

## Frozen inputs and causal separation

Each case is a four-second, 640 × 360, 30 fps video: 120 source frames, 720 frames
and 24 seconds across the batch. One 80 × 64 pixel textured rectangle translates
over a uniform background. The seed is manually authored inside that rectangle
on the first frame. Its center is the image anchor; it is not a measured physical
centroid. The low-texture negative uses an untextured rectangle. The occlusion
case removes the rectangle for frames 45–64 while its authored trajectory
continues.

The deterministic generator freezes trajectories, encoded source hashes and
actual frame presentation timestamps before tracking. Reference timestamps come
from integer video PTS multiplied by its rational time base. The tracker receives
only the manifest, initial seed and processing cadence. Future reference
positions are opened by the evaluator after processing and checked against their
frozen hashes. No future reference, simulator state or video frame enters an
earlier motion estimate or forecast.

The tracker reports the initial seed center plus accumulated robust feature
translation. Its current estimate is the observed anchor in this unsmoothed
baseline. Velocity uses two contiguous actual observations and their actual
time difference. A lost track stays lost, with no replacement, interpolation or
automatic reacquisition. Missing evidence becomes an abstention.

## Tracking results

| Authored case | Tracked / visible reference frames | Visible track coverage | Anchor error p95, px | Velocity error p95, px/s |
| --- | ---: | ---: | ---: | ---: |
| Stationary | 120 / 120 | 100% | 0 | 0 |
| Constant velocity | 120 / 120 | 100% | 0.00691 | 0.0574 |
| Stop/start | 120 / 120 | 100% | 0.00698 | 0.0621 |
| Direction change | 120 / 120 | 100% | 0.00714 | 0.0627 |
| Brief occlusion | 45 / 100 | 45% | 0.00722 | 0.0582 |
| Low texture | 0 / 120 | 0% | Unavailable | Unavailable |

All 720 source frames have attempted observation rows. All four textured cases
without occlusion retained their anchors. The occlusion case lost the track
when the object disappeared and remained lost after reappearance: 75 lost rows,
including 55 visible rows. The low-texture case abstained on all 120 frames.
Neither negative case is a perfect prediction result; the accuracy sample
counts become zero where no valid anchor/forecast exists. Velocity scores
exclude the two-frame neighborhood of authored speed discontinuities.

## Forecast results and failure tails

Errors are image-plane endpoint distances in pixels. Hold-current and
constant-velocity forecasts below share the same evaluable current anchors,
reference targets and sample counts. Scores are conditional on producing a
forecast; their answer coverage is reported separately.

| Authored case | 100 ms CV mean / hold mean, px | 500 ms CV mean / hold mean, px | 500 ms CV p95 / worst, px | Forecast samples at 100 / 500 ms |
| --- | ---: | ---: | ---: | ---: |
| Stationary | 0 / 0 | 0 / 0 | 0 / 0 | 116 / 104 |
| Constant velocity | 0.00675 / 9.005 | 0.0252 / 45.005 | 0.0337 / 0.0914 | 116 / 104 |
| Stop/start | 0.317 / 5.903 | 6.937 / 27.699 | 38.531 / 45.007 | 116 / 104 |
| Direction change | 0.317 / 8.897 | 6.945 / 41.769 | 59.109 / 89.975 | 116 / 104 |
| Brief occlusion | 0.00664 / 9.006 | 0.0252 / 45.006 | 0.0342 / 0.0914 | 44 / 44 |
| Low texture | Unavailable | Unavailable | Unavailable | 0 / 0 |

Every case has 117 available future reference frames at 100 ms and 105 at
500 ms. Terminal forecasts are censored: three and fifteen rows respectively.
The first tracked row has no velocity and therefore no forecast. The four
continuous textured cases answer on 116/117 and 104/105 available targets.
Occlusion answers on 44/117 and 44/105; low texture answers on zero. The remaining
rows are explicit abstentions, not zero-error samples. All 250 ms metrics and
denominators are also retained in the result file.

Current anchors match the same acquisition PTS within a 2 μs numeric
serialization tolerance. Off-grid observations cannot borrow nearby reference
truth. Future targets use the nearest actual source frame within half a frame
(1/60 second), with the earlier frame chosen on an exact tie. The 250 ms horizon
lands between 30 Hz frames; its reference is therefore approximately 16.67 ms
from the nominal target. The constant-velocity case consequently has about
1.49 px mean error there even though its 100/500 ms error is much smaller.
These are nearest-frame endpoint scores, with offsets disclosed, rather than
exact-time 250 ms accuracy. No backward interpolation produces new references.

The worst direction-change forecast misses by approximately 90 px at 500 ms.
Its low whole-case mean does not make it reliable at the turn. Passive
extrapolation cannot know an unannounced future reversal or contact action.
Interaction-conditioned models and evaluation slices remain subsequent work.
The propagated uncertainty radii are heuristic and uncalibrated; their empirical
coverage and size are recorded without probability claims.

## Processing and reproducibility

Six of six attempted cases completed with no pipeline failures. Processing took
**1.454 seconds**, plus **1.514 seconds** for fixture generation. The offline
batch-time/source-duration ratio is **0.0606**. Processing includes intake,
probing, streaming decode, causal tracking/forecast, writes and scoring; it
excludes fixture generation and provisioning. The ratio is withheld for any
failed batch so early failure cannot appear to improve throughput.

This was one run on Darwin 25.5.0, ARM64, ten logical CPUs, Python 3.12.9,
OpenCV 5.0.0 and NumPy 2.5.3. One OpenCV thread was requested, while OpenCV
reported ten. Actual peak thread concurrency, enforcement of the requested
limit and a minimum hardware requirement are unmeasured.
The process-lifetime resident high-water mark was **83,591,168 bytes
(79.72 MiB)**, including fixture generation; it is not a per-stage limit.
No GPU or paid inference API is used in this baseline. A fast batch result does
not establish live capture-to-action latency, worst-case deadlines, long-running
stability or robot-control benefit.

From an installed checkout containing the motion alpha, with the observations
extra and FFmpeg/ffprobe installed:

```sh
dvidia-motion benchmark --output runs/my-motion-benchmark
dvidia-motion inspect runs/my-motion-benchmark
dvidia-motion studio runs/my-motion-benchmark --port 8286
```

Use a new output directory each time. Complete local replay bundles retain
generated videos, actual-clock frame metadata, causal rows, seed evidence and
all attempted receipts. This public aggregate is not itself a replay bundle.
Generator and implementation hashes bind the protocol; encoder/platform
differences may change encoded-file hashes, so compare declared software and
parameters before interpreting reproduction differences.

These six cases share one authoring recipe and texture. They are engineering
checks rather than six independent real-world task attempts or a held-out
generalization cohort. Next evidence should include unseen permitted footage,
camera movement, blur, deformation and separate live stream measurements.
1 change: 1 addition & 0 deletions benchmarks/motion-alpha-20261010/protocol.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"baseline_comparison":"hold current on the identical evaluable anchored rows with a constant-velocity forecast","cases":["stationary","constant_velocity","stop_start","direction_change","brief_occlusion","low_texture"],"claim_scope":"authored synthetic image tracking and passive forecasting; no detection, real-world generalization or robot control","created_at":"2026-10-11T01:52:48.199306+00:00","fixture_recipe":{"background_rgb":[230,230,230],"encoder":"libx264 fast crf18 yuv420p single-thread; no metadata","fps":30,"frames":120,"height":360,"object_size_pixels":[80,64],"seed_inset_pixels":4,"texture_rng":"numpy.default_rng(4702)","trajectory_code_sha256":"d371dcabd905948670e9d9e49bdb4c2c7abb3f8a761d4daf33b68ca73e72919d","width":640},"fixtures":[{"frame_count":120,"id":"stationary","raw_rgb_bytes":82944000,"raw_rgb_sha256":"f4eea8057b6a26f818e06938a1c668bccb506455baedaa92c7a66c4d909df07f","reference_path":"stationary.reference.json","reference_sha256":"e53dbab795cd350e7d9aa6c185a75c4568083cf2a95f020fb62a4f233cb727a3","seed_box_xyxy_normalized":[0.19375,0.4222222222222222,0.30625,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":10152,"video_path":"stationary.mp4","video_sha256":"661290c74c0e3a7bede5a1290dcd7ead9f72a3ad3b834e38c36fe2d7d26dc777"},{"frame_count":120,"id":"constant_velocity","raw_rgb_bytes":82944000,"raw_rgb_sha256":"18331c7f1e2aa2d24cb11bed156e4b195ce0c3d62e5e31372af28e4bacec5758","reference_path":"constant_velocity.reference.json","reference_sha256":"f6f2ef67f55c901a1d1e00b6cf53884df1231e25cccdc386e3e0370e89c7ec86","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":26211,"video_path":"constant_velocity.mp4","video_sha256":"6534e4b76997dce97ce5db45727aae5b2317ca9d4a2ca4cd0b64117de7777f24"},{"frame_count":120,"id":"stop_start","raw_rgb_bytes":82944000,"raw_rgb_sha256":"f2848c6e07955ece5034c4fb8dcf3ab03d1bd7b95b87fa08b92774c6c7889ad2","reference_path":"stop_start.reference.json","reference_sha256":"3dcf4bd249a915e2bd1f0c36d65e7b818c0eadfe4881aaa1fa3e6d761a5407c1","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":23690,"video_path":"stop_start.mp4","video_sha256":"7ff0245ab68b43158888f00938352dceda59f23da4fff69c41e0ceaccb2fff5c"},{"frame_count":120,"id":"direction_change","raw_rgb_bytes":82944000,"raw_rgb_sha256":"746ae09cc463a5a65ed2b91c477daf968254a8b48896634ce6aa26f75d633db7","reference_path":"direction_change.reference.json","reference_sha256":"e3914ebbc1f494bd8babbb0b03913230c27a10839a46a5e1f2bb6f566611e9ce","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":33361,"video_path":"direction_change.mp4","video_sha256":"db277f4b419a4ac7d07ecbf7be2c8716c4782ca9541795ac6f3ce663c2e01fd7"},{"frame_count":120,"id":"brief_occlusion","raw_rgb_bytes":82944000,"raw_rgb_sha256":"3acfa2a20af83918a939c649b58e6cb1e6b45728c617e06e399b7502fa663283","reference_path":"brief_occlusion.reference.json","reference_sha256":"6f328d0f73683ab5076a98f0d3a82949dbd5f19b5959317b520ac42563ff5727","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":31840,"video_path":"brief_occlusion.mp4","video_sha256":"6ed24e1feb03789cd4af64a6c1d9c95a91c10bc7c85081b24c93b875ff5f8815"},{"frame_count":120,"id":"low_texture","raw_rgb_bytes":82944000,"raw_rgb_sha256":"fe4c0c3422e76dcea4d89f833d09933c5088a50cfba6b5cb745aaec32a57768a","reference_path":"low_texture.reference.json","reference_sha256":"e53dbab795cd350e7d9aa6c185a75c4568083cf2a95f020fb62a4f233cb727a3","seed_box_xyxy_normalized":[0.19375,0.4222222222222222,0.30625,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":5543,"video_path":"low_texture.mp4","video_sha256":"4eeb0a63d2cfc7d4f09b6079f2f8c9ab90af61a37c1d90f6decaedd34a455fd9"}],"hardware":{"logical_cpu_count":10,"machine":"arm64","os_release":"25.5.0","processor":"arm","python":"3.12.9","system":"Darwin"},"horizons_seconds":[0.1,0.25,0.5],"kind":"dvidia.motion-benchmark-protocol.v1","reference_access":"scorer only; never a tracker argument","sample_hz":30,"software":{"ffmpeg":"ffmpeg version 9.0.1 Copyright (c) 2000-2026 the FFmpeg developers","ffprobe":"ffprobe version 9.0.1 Copyright (c) 2007-2026 the FFmpeg developers"},"sources_manifest_sha256":"329c65421a25cf73e70f55603ed78ad6ec55aae707dfc3d990c8868a4f807ae3"}
1 change: 1 addition & 0 deletions benchmarks/motion-alpha-20261010/result.json

Large diffs are not rendered by default.

Loading
Loading