Skip to content

Commit ade2b54

Browse files
author
DVIDIA
committed
Add causal image-motion alpha, forecast replay and measured benchmarks
1 parent 9fdf2ad commit ade2b54

21 files changed

Lines changed: 3740 additions & 3 deletions

‎.github/workflows/ci.yml‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,42 @@ jobs:
6565
dvidia-motor-memory --help
6666
dvidia-observe --help
6767
dvidia-observation-studio --help
68+
dvidia-motion --help
6869
dvidia-training-doctor --json
6970
- name: Test footage, visual training and loopback studio
70-
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models -v
71+
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models tests.test_motion tests.test_motion_benchmark -v
72+
73+
motion:
74+
name: Causal image motion / Linux / Python 3.12
75+
runs-on: ubuntu-latest
76+
timeout-minutes: 15
77+
steps:
78+
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
79+
with:
80+
persist-credentials: false
81+
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
82+
with:
83+
python-version: "3.12"
84+
- name: Install CPU optical-flow runtime
85+
run: |
86+
sudo apt-get update
87+
sudo apt-get install -y ffmpeg
88+
python -m pip install ".[observations]"
89+
python -m pip check
90+
dvidia-motion --help
91+
- name: Verify source clocks, causal rows, scoring and local replay
92+
run: python -m unittest tests.test_motion tests.test_motion_video tests.test_motion_benchmark tests.test_motion_studio -v
93+
- name: Run and inspect authored tracking and prediction fixtures
94+
run: |
95+
dvidia-motion benchmark --output runs/image-motion-ci
96+
dvidia-motion inspect runs/image-motion-ci
97+
- name: Preserve every attempted synthetic motion case
98+
if: always()
99+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
100+
with:
101+
name: authored-image-motion
102+
path: runs/image-motion-ci
103+
if-no-files-found: ignore
71104

72105
observations:
73106
name: Observation detectors / Linux / Python 3.12

‎README.md‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,31 @@ six public excerpts. This establishes the integration, not accuracy: labels can
5858
be wrong, outcomes remain unknown, and there is no 3D reconstruction, automatic
5959
action understanding, or human-video-to-robot control bridge.
6060

61+
## Estimate and forecast image motion
62+
63+
The [first image-motion alpha](docs/motion-observations.md) is available in the
64+
candidate source checkout. Manually seed one first-frame region; causal CPU
65+
feature tracking estimates its image-anchor speed and forecasts a path at
66+
100, 250 and 500 ms. Lost tracks stay lost. Actual integer video timestamps,
67+
tracking support, units, source/implementation hashes and failed attempts are
68+
preserved. This feature trains no model and requires no detector weights.
69+
70+
```sh
71+
.venv/bin/python -m pip install '.[observations]'
72+
.venv/bin/dvidia-motion benchmark --output runs/image-motion-001
73+
.venv/bin/dvidia-motion inspect runs/image-motion-001
74+
.venv/bin/dvidia-motion studio runs/image-motion-001 --port 8286
75+
```
76+
77+
Use the [candidate source installation instructions](docs/motion-observations.md#install-from-source),
78+
not the frozen `observation-lab-v0.1.0` tag, for `dvidia-motion`. The authored
79+
benchmark and read-only local replay test image tracking and prediction, not
80+
real detector accuracy. Speeds are analysis-image pixels/second; physical speed,
81+
moving-camera compensation, live robot latency and interaction prediction remain
82+
unqualified. This is one component toward [the full Observation Lab v0.2](docs/observation-v0.2-plan.md).
83+
See the [measured authored benchmark](benchmarks/motion-alpha-20261010/README.md),
84+
including conditional errors, failed tracking and forecast tails.
85+
6186
## Try the pipeline
6287

6388
Generate an explicitly synthetic example, then train and verify its output:
@@ -132,4 +157,7 @@ Each command uses a fresh output directory where applicable. The model and compl
132157

133158
The demo presents saved candidate and open-jaw control playback from one exact simulation scene. Training runs locally; the dataset and model cards document reproducible inputs, CPU measurements and the limits of this research alpha. [Maintainer publishing tools](publishing/huggingface/README.md) stage only the already-public synthetic release and use a credential from macOS Keychain without printing or caching it.
134159

135-
MIT licensed. Research and measured limitations: [research.dvidia.org](https://research.dvidia.org). Contact: [hello@dvidia.org](mailto:hello@dvidia.org).
160+
Original DVIDIA code is MIT licensed; observation detector adaptations retain
161+
Apache 2.0 notices and upstream models retain their own licenses. Input footage
162+
has separate rights. Research and measured limitations: [research.dvidia.org](https://research.dvidia.org).
163+
Contact: [hello@dvidia.org](mailto:hello@dvidia.org).
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# DVIDIA image-motion alpha — 10 October 2026
2+
3+
**Six authored fixtures processed successfully; real-world accuracy and live
4+
robot latency remain unmeasured.** This evaluates sparse tracking of a manually
5+
seeded image region and passive constant-velocity forecasting. It evaluates
6+
neither an object detector nor a trained interaction or robot-control model.
7+
8+
The [frozen protocol](protocol.json) and [aggregate results](result.json)
9+
preserve parameters, input/reference hashes, software, attempted counts,
10+
conditional errors, abstentions and timing scope. They contain no input footage,
11+
model weights, private data or credentials. The generated fixtures and these
12+
DVIDIA artifacts are MIT licensed. The existing project notices continue to
13+
apply to upstream dependencies.
14+
15+
## Frozen inputs and causal separation
16+
17+
Each case is a four-second, 640 × 360, 30 fps video: 120 source frames, 720 frames
18+
and 24 seconds across the batch. One 80 × 64 pixel textured rectangle translates
19+
over a uniform background. The seed is manually authored inside that rectangle
20+
on the first frame. Its center is the image anchor; it is not a measured physical
21+
centroid. The low-texture negative uses an untextured rectangle. The occlusion
22+
case removes the rectangle for frames 45–64 while its authored trajectory
23+
continues.
24+
25+
The deterministic generator freezes trajectories, encoded source hashes and
26+
actual frame presentation timestamps before tracking. Reference timestamps come
27+
from integer video PTS multiplied by its rational time base. The tracker receives
28+
only the manifest, initial seed and processing cadence. Future reference
29+
positions are opened by the evaluator after processing and checked against their
30+
frozen hashes. No future reference, simulator state or video frame enters an
31+
earlier motion estimate or forecast.
32+
33+
The tracker reports the initial seed center plus accumulated robust feature
34+
translation. Its current estimate is the observed anchor in this unsmoothed
35+
baseline. Velocity uses two contiguous actual observations and their actual
36+
time difference. A lost track stays lost, with no replacement, interpolation or
37+
automatic reacquisition. Missing evidence becomes an abstention.
38+
39+
## Tracking results
40+
41+
| Authored case | Tracked / visible reference frames | Visible track coverage | Anchor error p95, px | Velocity error p95, px/s |
42+
| --- | ---: | ---: | ---: | ---: |
43+
| Stationary | 120 / 120 | 100% | 0 | 0 |
44+
| Constant velocity | 120 / 120 | 100% | 0.00691 | 0.0574 |
45+
| Stop/start | 120 / 120 | 100% | 0.00698 | 0.0621 |
46+
| Direction change | 120 / 120 | 100% | 0.00714 | 0.0627 |
47+
| Brief occlusion | 45 / 100 | 45% | 0.00722 | 0.0582 |
48+
| Low texture | 0 / 120 | 0% | Unavailable | Unavailable |
49+
50+
All 720 source frames have attempted observation rows. All four textured cases
51+
without occlusion retained their anchors. The occlusion case lost the track
52+
when the object disappeared and remained lost after reappearance: 75 lost rows,
53+
including 55 visible rows. The low-texture case abstained on all 120 frames.
54+
Neither negative case is a perfect prediction result; the accuracy sample
55+
counts become zero where no valid anchor/forecast exists. Velocity scores
56+
exclude the two-frame neighborhood of authored speed discontinuities.
57+
58+
## Forecast results and failure tails
59+
60+
Errors are image-plane endpoint distances in pixels. Hold-current and
61+
constant-velocity forecasts below share the same evaluable current anchors,
62+
reference targets and sample counts. Scores are conditional on producing a
63+
forecast; their answer coverage is reported separately.
64+
65+
| Authored case | 100 ms CV mean / hold mean, px | 500 ms CV mean / hold mean, px | 500 ms CV p95 / worst, px | Forecast samples at 100 / 500 ms |
66+
| --- | ---: | ---: | ---: | ---: |
67+
| Stationary | 0 / 0 | 0 / 0 | 0 / 0 | 116 / 104 |
68+
| Constant velocity | 0.00675 / 9.005 | 0.0252 / 45.005 | 0.0337 / 0.0914 | 116 / 104 |
69+
| Stop/start | 0.317 / 5.903 | 6.937 / 27.699 | 38.531 / 45.007 | 116 / 104 |
70+
| Direction change | 0.317 / 8.897 | 6.945 / 41.769 | 59.109 / 89.975 | 116 / 104 |
71+
| Brief occlusion | 0.00664 / 9.006 | 0.0252 / 45.006 | 0.0342 / 0.0914 | 44 / 44 |
72+
| Low texture | Unavailable | Unavailable | Unavailable | 0 / 0 |
73+
74+
Every case has 117 available future reference frames at 100 ms and 105 at
75+
500 ms. Terminal forecasts are censored: three and fifteen rows respectively.
76+
The first tracked row has no velocity and therefore no forecast. The four
77+
continuous textured cases answer on 116/117 and 104/105 available targets.
78+
Occlusion answers on 44/117 and 44/105; low texture answers on zero. The remaining
79+
rows are explicit abstentions, not zero-error samples. All 250 ms metrics and
80+
denominators are also retained in the result file.
81+
82+
Current anchors match the same acquisition PTS within a 2 μs numeric
83+
serialization tolerance. Off-grid observations cannot borrow nearby reference
84+
truth. Future targets use the nearest actual source frame within half a frame
85+
(1/60 second), with the earlier frame chosen on an exact tie. The 250 ms horizon
86+
lands between 30 Hz frames; its reference is therefore approximately 16.67 ms
87+
from the nominal target. The constant-velocity case consequently has about
88+
1.49 px mean error there even though its 100/500 ms error is much smaller.
89+
These are nearest-frame endpoint scores, with offsets disclosed, rather than
90+
exact-time 250 ms accuracy. No backward interpolation produces new references.
91+
92+
The worst direction-change forecast misses by approximately 90 px at 500 ms.
93+
Its low whole-case mean does not make it reliable at the turn. Passive
94+
extrapolation cannot know an unannounced future reversal or contact action.
95+
Interaction-conditioned models and evaluation slices remain subsequent work.
96+
The propagated uncertainty radii are heuristic and uncalibrated; their empirical
97+
coverage and size are recorded without probability claims.
98+
99+
## Processing and reproducibility
100+
101+
Six of six attempted cases completed with no pipeline failures. Processing took
102+
**1.454 seconds**, plus **1.514 seconds** for fixture generation. The offline
103+
batch-time/source-duration ratio is **0.0606**. Processing includes intake,
104+
probing, streaming decode, causal tracking/forecast, writes and scoring; it
105+
excludes fixture generation and provisioning. The ratio is withheld for any
106+
failed batch so early failure cannot appear to improve throughput.
107+
108+
This was one run on Darwin 25.5.0, ARM64, ten logical CPUs, Python 3.12.9,
109+
OpenCV 5.0.0 and NumPy 2.5.3. One OpenCV thread was requested, while OpenCV
110+
reported ten. Actual peak thread concurrency, enforcement of the requested
111+
limit and a minimum hardware requirement are unmeasured.
112+
The process-lifetime resident high-water mark was **83,591,168 bytes
113+
(79.72 MiB)**, including fixture generation; it is not a per-stage limit.
114+
No GPU or paid inference API is used in this baseline. A fast batch result does
115+
not establish live capture-to-action latency, worst-case deadlines, long-running
116+
stability or robot-control benefit.
117+
118+
From an installed checkout containing the motion alpha, with the observations
119+
extra and FFmpeg/ffprobe installed:
120+
121+
```sh
122+
dvidia-motion benchmark --output runs/my-motion-benchmark
123+
dvidia-motion inspect runs/my-motion-benchmark
124+
dvidia-motion studio runs/my-motion-benchmark --port 8286
125+
```
126+
127+
Use a new output directory each time. Complete local replay bundles retain
128+
generated videos, actual-clock frame metadata, causal rows, seed evidence and
129+
all attempted receipts. This public aggregate is not itself a replay bundle.
130+
Generator and implementation hashes bind the protocol; encoder/platform
131+
differences may change encoded-file hashes, so compare declared software and
132+
parameters before interpreting reproduction differences.
133+
134+
These six cases share one authoring recipe and texture. They are engineering
135+
checks rather than six independent real-world task attempts or a held-out
136+
generalization cohort. Next evidence should include unseen permitted footage,
137+
camera movement, blur, deformation and separate live stream measurements.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"baseline_comparison":"hold current on the identical evaluable anchored rows with a constant-velocity forecast","cases":["stationary","constant_velocity","stop_start","direction_change","brief_occlusion","low_texture"],"claim_scope":"authored synthetic image tracking and passive forecasting; no detection, real-world generalization or robot control","created_at":"2026-10-11T01:52:48.199306+00:00","fixture_recipe":{"background_rgb":[230,230,230],"encoder":"libx264 fast crf18 yuv420p single-thread; no metadata","fps":30,"frames":120,"height":360,"object_size_pixels":[80,64],"seed_inset_pixels":4,"texture_rng":"numpy.default_rng(4702)","trajectory_code_sha256":"d371dcabd905948670e9d9e49bdb4c2c7abb3f8a761d4daf33b68ca73e72919d","width":640},"fixtures":[{"frame_count":120,"id":"stationary","raw_rgb_bytes":82944000,"raw_rgb_sha256":"f4eea8057b6a26f818e06938a1c668bccb506455baedaa92c7a66c4d909df07f","reference_path":"stationary.reference.json","reference_sha256":"e53dbab795cd350e7d9aa6c185a75c4568083cf2a95f020fb62a4f233cb727a3","seed_box_xyxy_normalized":[0.19375,0.4222222222222222,0.30625,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":10152,"video_path":"stationary.mp4","video_sha256":"661290c74c0e3a7bede5a1290dcd7ead9f72a3ad3b834e38c36fe2d7d26dc777"},{"frame_count":120,"id":"constant_velocity","raw_rgb_bytes":82944000,"raw_rgb_sha256":"18331c7f1e2aa2d24cb11bed156e4b195ce0c3d62e5e31372af28e4bacec5758","reference_path":"constant_velocity.reference.json","reference_sha256":"f6f2ef67f55c901a1d1e00b6cf53884df1231e25cccdc386e3e0370e89c7ec86","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":26211,"video_path":"constant_velocity.mp4","video_sha256":"6534e4b76997dce97ce5db45727aae5b2317ca9d4a2ca4cd0b64117de7777f24"},{"frame_count":120,"id":"stop_start","raw_rgb_bytes":82944000,"raw_rgb_sha256":"f2848c6e07955ece5034c4fb8dcf3ab03d1bd7b95b87fa08b92774c6c7889ad2","reference_path":"stop_start.reference.json","reference_sha256":"3dcf4bd249a915e2bd1f0c36d65e7b818c0eadfe4881aaa1fa3e6d761a5407c1","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":23690,"video_path":"stop_start.mp4","video_sha256":"7ff0245ab68b43158888f00938352dceda59f23da4fff69c41e0ceaccb2fff5c"},{"frame_count":120,"id":"direction_change","raw_rgb_bytes":82944000,"raw_rgb_sha256":"746ae09cc463a5a65ed2b91c477daf968254a8b48896634ce6aa26f75d633db7","reference_path":"direction_change.reference.json","reference_sha256":"e3914ebbc1f494bd8babbb0b03913230c27a10839a46a5e1f2bb6f566611e9ce","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":33361,"video_path":"direction_change.mp4","video_sha256":"db277f4b419a4ac7d07ecbf7be2c8716c4782ca9541795ac6f3ce663c2e01fd7"},{"frame_count":120,"id":"brief_occlusion","raw_rgb_bytes":82944000,"raw_rgb_sha256":"3acfa2a20af83918a939c649b58e6cb1e6b45728c617e06e399b7502fa663283","reference_path":"brief_occlusion.reference.json","reference_sha256":"6f328d0f73683ab5076a98f0d3a82949dbd5f19b5959317b520ac42563ff5727","seed_box_xyxy_normalized":[0.1,0.4222222222222222,0.2125,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":31840,"video_path":"brief_occlusion.mp4","video_sha256":"6ed24e1feb03789cd4af64a6c1d9c95a91c10bc7c85081b24c93b875ff5f8815"},{"frame_count":120,"id":"low_texture","raw_rgb_bytes":82944000,"raw_rgb_sha256":"fe4c0c3422e76dcea4d89f833d09933c5088a50cfba6b5cb745aaec32a57768a","reference_path":"low_texture.reference.json","reference_sha256":"e53dbab795cd350e7d9aa6c185a75c4568083cf2a95f020fb62a4f233cb727a3","seed_box_xyxy_normalized":[0.19375,0.4222222222222222,0.30625,0.5777777777777777],"source_duration_seconds":4.0,"video_bytes":5543,"video_path":"low_texture.mp4","video_sha256":"4eeb0a63d2cfc7d4f09b6079f2f8c9ab90af61a37c1d90f6decaedd34a455fd9"}],"hardware":{"logical_cpu_count":10,"machine":"arm64","os_release":"25.5.0","processor":"arm","python":"3.12.9","system":"Darwin"},"horizons_seconds":[0.1,0.25,0.5],"kind":"dvidia.motion-benchmark-protocol.v1","reference_access":"scorer only; never a tracker argument","sample_hz":30,"software":{"ffmpeg":"ffmpeg version 9.0.1 Copyright (c) 2000-2026 the FFmpeg developers","ffprobe":"ffprobe version 9.0.1 Copyright (c) 2007-2026 the FFmpeg developers"},"sources_manifest_sha256":"329c65421a25cf73e70f55603ed78ad6ec55aae707dfc3d990c8868a4f807ae3"}

‎benchmarks/motion-alpha-20261010/result.json‎

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)