Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 44 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,52 @@ jobs:
dvidia-episode-tape --help
dvidia-reaction --help
dvidia-motor-memory --help
dvidia-observe --help
dvidia-observation-studio --help
dvidia-training-doctor --json
- name: Test footage, visual training and loopback studio
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark -v
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models -v

observations:
name: Observation detectors / Linux / Python 3.12
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
with:
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.12"
- name: Install observation runtime
run: |
sudo apt-get update
sudo apt-get install -y ffmpeg
python -m pip install ".[observations]"
python -m pip check
- name: Verify review contracts and explicitly provision pinned detectors
run: |
python -m unittest tests.test_observations tests.test_observation_models -v
python -m unittest discover -s publishing/huggingface -p 'test_publish_observation.py' -v
python -m unittest discover -s publishing/huggingface/observation-lab -p 'test_*.py' -v
node publishing/huggingface/observation-lab/test_adapter.cjs
python scripts/download_observation_models.py --download --output runs/observation-models-ci
- name: Smoke test actual ONNX loading and inference
run: |
python - <<'PY'
import json, tempfile
from pathlib import Path
import cv2, numpy as np
from dvidia_training.observation_models import OpenCVDetector
detector = OpenCVDetector('runs/observation-models-ci')
with tempfile.TemporaryDirectory() as directory:
frame = Path(directory) / 'neutral.png'
assert cv2.imwrite(str(frame), np.full((240, 320, 3), 240, np.uint8))
rows = detector.detect(frame)
assert isinstance(rows, list)
print(json.dumps({'scope': 'neutral-image model-loading smoke test; no accuracy measurement',
'proposals': len(rows), 'runtime': detector.metadata}))
PY

arm:
name: Optional arm / Linux / Python 3.12
Expand Down
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,24 @@ Open [the collection planner](http://127.0.0.1:8270/coverage) in the same local

The [episode tape recorder](docs/episode-tape.md) preserves the existing simulated arm's measured state, requested/applied commands, contacts and sensor ages. Recording needs the optional arm extra; validation and inspection need only Python's standard library. These tools are available from the current checkout and do not train a new policy or change the older Hugging Face pilot.

## Review what is visible in a video

The new [local observation lab](docs/observations.md) uses optional CPU object and
palm detectors to propose timestamped image observations. Open the original video,
inspect its boxes and timeline, correct labels, and save a separate review revision.
Raw evidence stays unchanged. No account, cloud inference, or GPU is needed.

[Observation Lab release](https://github.com/Dvidia-Inference/dvidia-training/releases/tag/observation-lab-v0.1.0) · [Research paper](https://research.dvidia.org/papers/dvidia-observation-lab/) · [Synthetic review demo](https://huggingface.co/spaces/Dvidia/observation-lab)

Use the `observation-lab-v0.1.0` tag for this feature. The hosted demo uses
authored synthetic clips and keeps edits in your browser; actual detection runs
locally with the optional observation runtime and explicitly provisioned models.

The [first feasibility run](benchmarks/observation-20261010/README.md) completed
six public excerpts. This establishes the integration, not accuracy: labels can
be wrong, outcomes remain unknown, and there is no 3D reconstruction, automatic
action understanding, or human-video-to-robot control bridge.

## Try the pipeline

Generate an explicitly synthetic example, then train and verify its output:
Expand Down
103 changes: 103 additions & 0 deletions benchmarks/observation-20261010/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Local observation feasibility — 10 October 2026

**Engineering integration passed on six public excerpts; accuracy not assessed.**
This is not a cup-placement benchmark, 3D reconstruction or trained robot skill.
The [plan](../../docs/observation-pilot-plan.md) was written before the model run.

## Inputs and execution

The fixed order was shirt, stir, pack, onion, melon, jeans, selected from existing
local public examples before examining model output. Five are attributed
EgoAnnotate excerpts; stir is a CoMind excerpt. Licenses and public source links
are preserved in both protocol files. Exact excerpts were hash-verified. The
full upstream originals were not downloaded or hash-verified for this pilot.
No raw media, sampled frames or private user metadata are included in this report.

Six excerpts total 19 seconds, 2,753,287 bytes and 38 sampled frames. They are not
six independent complete attempts; unknown session grouping remains one
conservative shared group. No train/test split or learning run took place.
Publisher titles were never fed to the detector.

| Attempt | Complete / attempted | Processing time | Failures |
| --- | --- | ---: | ---: |
| Initial recipe v1 | 6 / 6 | 4.835 seconds | 0 |
| Hardened recipe v1.1 | 6 / 6 | 4.635 seconds | 0 |

Timing covers intake, copy/hash, frame decoding, detection, association and
episode writes. It excludes model provisioning/loading, final receipt writing,
UI startup and human review. Single runs, warm machine; this is neither a p95
latency measurement nor a production throughput guarantee. Peak RAM, energy and
dollar cost were not measured. No paid inference API was called.

Runtime: Python 3.12.9, OpenCV 5.0.0.93, NumPy 2.5.3, macOS ARM64 and one
inference worker. Four OpenCV threads were requested, but the saved model
metadata records `cv2.getNumThreads()` as ten after model loading. Actual peak
thread concurrency was not measured; a four-thread cap is not established.
Both pinned ONNX models total approximately 40 MB. OpenCV
5 printed that target selection is unsupported by its new graph engine; this
build uses the OpenCV CPU backend and no GPU path was requested or measured.

## What changed after the first run

Review identified a possible browser-clock shift for videos with nonzero start
timestamps. Recipe v1.1 rejects nonzero container/video/decoded starts rather
than moving boxes relative to the unchanged original. The initial artifacts are
preserved alongside the subsequent run, which processed the same six inputs.
All six inputs have zero starts and were accepted in both runs.

The review server was also hardened against same-size media replacement,
malformed review IDs and oversized run metadata. These are workflow fixes,
not improvements in detector quality.

## Observed limitations

- The three-frame preliminary adapter check included a saucepan incorrectly
labeled cup and storage jars labeled cups. Hands were missed in the stir
sample. These are qualitative findings, not scored accuracy estimates.
- COCO labels do not cover every task object; palm rectangles do not cover full
hands or prove contact. Two frames/second can miss short actions.
- Association uses same-class image overlap. It is not optical flow and can
fragment or switch identities. Overlap proposals do not establish grasping.
- Outcomes remain unknown. Model scores are uncalibrated, and the pilot creates
no automatic natural-language action captions or robot commands.
- The intended permission-cleared cup-placement cohort and independent reference
annotations have not been collected. Accuracy, false-completion rate and
correction time remain unmeasured. No model ranking is established.

## Evidence

- `initial-protocol.json`, `initial-run.json`: original frozen inputs and receipt.
- `hardened-protocol.json`, `hardened-run.json`: later timing policy and receipt.
- `episode-summary.json`: durations, proposal counts, timing and excerpt hashes.
- Local ignored run directories retain media, frames and raw episode annotations.
Run and source hashes bind the receipts to those local artifacts; this report
alone cannot reproduce a visual inspection without obtaining permitted clips.

Browser interaction testing uses a separate copied run. Test review edits do not
count as independent human annotations and do not alter the original run's
unreviewed status.

## Implementation validation

The repository suite passed 307 tests, followed by 24 focused observation tests
including four additional detector-admission/suppression tests. These exercise
real FFmpeg decoding, timestamps, source tampering, failed attempts, malformed
reviews, revision conflicts, local-origin enforcement and video byte ranges.

Fifteen browser checks passed on an isolated review copy: playback, timestamped
overlays, correction save/reload, preservation of raw labels, manual events,
export, unsaved-edit protection, completion validation and 390/320px layouts.
There were no console/page errors. See `ui-validation.json`. Headless browser
checks are not physical-phone or Safari qualification.

The wheel built successfully; its final HTML, command entry points and both
upstream license files were verified. Raw footage/model files are Git-ignored.
Production uploads, accounts, storage and website deployment were not modified.

## Next decision

Collect 30–50 complete cup-placement attempts with permission and explicit
session/person grouping, including failures, occlusion and recovery. Establish
blinded reference labels first. Then compare object-only, temporal caption-only,
combined proposals and manual annotation at matched quality and total cost.
Only after that should an app integration claim reliable task labeling.
80 changes: 80 additions & 0 deletions benchmarks/observation-20261010/episode-summary.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
[
{
"id": "shirt",
"duration_seconds": 3.0,
"metrics": {
"accuracy_status": "not_assessed",
"detected_frames": 6,
"elapsed_seconds": 0.6693273328710347,
"frames": 6,
"inference_seconds": 0.46699687489308417
},
"proposals": 7,
"source_sha256": "8392c62f4e9590a15745311746114c2b6355e24c60fa87fef68e834f20762113"
},
{
"id": "stir",
"duration_seconds": 4.0,
"metrics": {
"accuracy_status": "not_assessed",
"detected_frames": 8,
"elapsed_seconds": 1.1173061658628285,
"frames": 8,
"inference_seconds": 0.6056890829931945
},
"proposals": 25,
"source_sha256": "4ffe78850d4e6475546a521e0eb425f7a34c632679183cf112aacb70ec1b0861"
},
{
"id": "pack",
"duration_seconds": 3.0,
"metrics": {
"accuracy_status": "not_assessed",
"detected_frames": 6,
"elapsed_seconds": 0.6751292911358178,
"frames": 6,
"inference_seconds": 0.4420628750231117
},
"proposals": 7,
"source_sha256": "87a67e5b10f124a65df7d669086d4a9b2cb7ae25a5d758594b9b3aa7b0938f4c"
},
{
"id": "onion",
"duration_seconds": 3.0,
"metrics": {
"accuracy_status": "not_assessed",
"detected_frames": 6,
"elapsed_seconds": 0.7398202081676573,
"frames": 6,
"inference_seconds": 0.43661770783364773
},
"proposals": 10,
"source_sha256": "100e25698ee99e99dc648c41d9864144e312f0fce7c07dc9a7b06b487b9a2a5f"
},
{
"id": "melon",
"duration_seconds": 3.0,
"metrics": {
"accuracy_status": "not_assessed",
"detected_frames": 6,
"elapsed_seconds": 0.7281540830153972,
"frames": 6,
"inference_seconds": 0.4237409171182662
},
"proposals": 14,
"source_sha256": "f32cf75d28602d169b4f302830ba633a351ac890596887c54c8e928efcf2650e"
},
{
"id": "jeans",
"duration_seconds": 3.0,
"metrics": {
"accuracy_status": "not_assessed",
"detected_frames": 6,
"elapsed_seconds": 0.6997090000659227,
"frames": 6,
"inference_seconds": 0.4299595837946981
},
"proposals": 9,
"source_sha256": "95845362951b3f78b7b731691df6d472bf176ced4b07ecef4d99d0b8a21da3e2"
}
]
1 change: 1 addition & 0 deletions benchmarks/observation-20261010/hardened-protocol.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"created_at":"2026-10-10T11:22:03.270961+00:00","execution":"local_offline","hardware":{"machine":"arm64","platform":"Darwin"},"kind":"dvidia.observation-protocol.v1","limits":{"cpu_threads":4,"workers":1},"manifest_sha256":"6e4e3a0b1b12eeb32cafcba0668d2614c80b022a91eafc7740018be880eddb29","model":{"backend":"opencv","coordinates":"normalized_image_xyxy","cpu_threads":10,"device":"cpu","hand_box_semantics":"Palm detector rectangle; not full hand segmentation or observed contact.","hand_detector_enabled":true,"hand_threshold":0.5,"implementation_sha256":"e1c6939779ab39a32497e00c35892a98eef59c3dd512135f1ac6cd4f897d1a22","implementation_version":1,"kind":"opencv-zoo-yolox-palm","limitations":["COCO labels can be wrong or absent, especially for small and occluded objects.","Palm boxes do not establish finger contact, grasp, force, depth, task success or robot actions.","Object identity and temporal relationships require a separate tracking stage."],"max_detections":30,"max_nms_candidates":1000,"models":[{"bytes":35858002,"directory":"object_detection_yolox","download_url":"https://media.githubusercontent.com/media/opencv/opencv_zoo/47534e27c9851bb1128ccc0102f1145e27f23f98/models/object_detection_yolox/object_detection_yolox_2022nov.onnx","license":"Apache-2.0","license_bytes":11371,"license_sha256":"0ec3668d3274bcf29e8a29e9576d5a2cd96fc78d3c5bec4387355a796e5d9088","license_url":"https://raw.githubusercontent.com/opencv/opencv_zoo/47534e27c9851bb1128ccc0102f1145e27f23f98/models/object_detection_yolox/LICENSE","name":"object_detection_yolox_2022nov.onnx","revision":"47534e27c9851bb1128ccc0102f1145e27f23f98","role":"objects","sha256":"c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063","source":"https://github.com/opencv/opencv_zoo/blob/47534e27c9851bb1128ccc0102f1145e27f23f98/models/object_detection_yolox/object_detection_yolox_2022nov.onnx"},{"bytes":3905734,"directory":"palm_detection_mediapipe","download_url":"https://media.githubusercontent.com/media/opencv/opencv_zoo/47534e27c9851bb1128ccc0102f1145e27f23f98/models/palm_detection_mediapipe/palm_detection_mediapipe_2023feb.onnx","license":"Apache-2.0","license_bytes":11358,"license_sha256":"cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30","license_url":"https://raw.githubusercontent.com/opencv/opencv_zoo/47534e27c9851bb1128ccc0102f1145e27f23f98/models/palm_detection_mediapipe/LICENSE","name":"palm_detection_mediapipe_2023feb.onnx","revision":"47534e27c9851bb1128ccc0102f1145e27f23f98","role":"palms","sha256":"78ff51c38496b7fc8b8ebdb6cc8c1abb02fa6c38427c6848254cdaba57fcce7c","source":"https://github.com/opencv/opencv_zoo/blob/47534e27c9851bb1128ccc0102f1145e27f23f98/models/palm_detection_mediapipe/palm_detection_mediapipe_2023feb.onnx"}],"nms_iou":{"objects":0.5,"palms":0.3},"runtime":{"numpy":"2.5.3","opencv":"5.0.0"},"score_kind":"uncalibrated_model_evidence","threshold":0.35},"recipe":{"automatic_outcome":"unknown","claim_scope":"sampled_2d_observations","id":"dvidia-observations-v1.1","max_duration_seconds":60,"max_frames":120,"max_image_side":640,"max_track_gap_seconds":0.8,"minimum_association_iou":0.2,"samples_per_second":2,"timebase":"zero-origin video and container; nonzero starts rejected"},"sources":[{"credit":"Taher Panbiharwala and Zainab Barwaniwala; EgoAnnotate v1; excerpt starts 0s, original full hash not verified","id":"shirt","kind":"public","license":"CC BY 4.0","permission":"local_analysis","session_id":"unknown-shared-publisher-session","sha256":"8392c62f4e9590a15745311746114c2b6355e24c60fa87fef68e834f20762113","source_recording_id":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010079.mp4","source_url":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010079.mp4","title":"Starting to fold a white shirt"},{"credit":"Alexey Gavryushin, Dingxi Zhang, Zhao Huang, Alexandros Delitzas, Jiaqi Chen, Ben Ellis, Cedric Z\u00f6llner, Manthan Patel, Manuel Kaufmann, Marc Pollefeys, and Xi Wang; CoMind (2026); excerpt starts 1s, original full hash not verified","id":"stir","kind":"public","license":"CC BY 4.0","permission":"local_analysis","session_id":"unknown-shared-publisher-session","sha256":"4ffe78850d4e6475546a521e0eb425f7a34c632679183cf112aacb70ec1b0861","source_recording_id":"https://comind.ethz.ch/","source_url":"https://comind.ethz.ch/","title":"Stirring vegetables in a pan"},{"credit":"Taher Panbiharwala and Zainab Barwaniwala; EgoAnnotate v1; excerpt starts 0s, original full hash not verified","id":"pack","kind":"public","license":"CC BY 4.0","permission":"local_analysis","session_id":"unknown-shared-publisher-session","sha256":"87a67e5b10f124a65df7d669086d4a9b2cb7ae25a5d758594b9b3aa7b0938f4c","source_recording_id":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010087.mp4","source_url":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010087.mp4","title":"Moving food from a tray toward a container"},{"credit":"Taher Panbiharwala and Zainab Barwaniwala; EgoAnnotate v1; excerpt starts 0s, original full hash not verified","id":"onion","kind":"public","license":"CC BY 4.0","permission":"local_analysis","session_id":"unknown-shared-publisher-session","sha256":"100e25698ee99e99dc648c41d9864144e312f0fce7c07dc9a7b06b487b9a2a5f","source_recording_id":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010072.mp4","source_url":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010072.mp4","title":"Slicing an onion on a pink plate"},{"credit":"Taher Panbiharwala and Zainab Barwaniwala; EgoAnnotate v1; excerpt starts 0s, original full hash not verified","id":"melon","kind":"public","license":"CC BY 4.0","permission":"local_analysis","session_id":"unknown-shared-publisher-session","sha256":"f32cf75d28602d169b4f302830ba633a351ac890596887c54c8e928efcf2650e","source_recording_id":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010076.mp4","source_url":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010076.mp4","title":"Cutting a watermelon wedge"},{"credit":"Taher Panbiharwala and Zainab Barwaniwala; EgoAnnotate v1; excerpt starts 0s, original full hash not verified","id":"jeans","kind":"public","license":"CC BY 4.0","permission":"local_analysis","session_id":"unknown-shared-publisher-session","sha256":"95845362951b3f78b7b731691df6d472bf176ced4b07ecef4d99d0b8a21da3e2","source_recording_id":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010081.mp4","source_url":"https://huggingface.co/datasets/TaherPanbiharwala/EgoAnnotate/blob/156d13adaee4110cf63e54fa940ec16ff199d12f/videos/GX010081.mp4","title":"Folding grey jeans"}]}
Loading
Loading