Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions docs/downloads/reports/Image goal Jev measured v02.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Jev image-goal verification: the measured follow-up

DVIDIA robotics research · 19 October 2026.

We ran all **64 frozen requests** against `jev-1.14.0`: 32 PiPER scenes, each evaluated once with images alone and once with privileged simulator measurements, in the prespecified shuffled order. Every request returned a valid structured response. No request was retried, and no fixture, prompt, threshold or scoring rule changed after the protocol was frozen.

**Jev made no false-completion declaration in this cohort, but it usually failed to recognize a genuinely finished task.** With images and telemetry it declared only **1/8 physically complete cases** complete. Images alone produced **0/8**. All four occluded finished scenes were called incomplete under both input conditions. The result is useful for designing the next experiment; it does not qualify Jev as a robot completion gate.

This is a measured follow-up to the unchanged [v0.1 protocol](https://research.dvidia.org/papers/image-goal-jev-v01/). The `v02` publication path records the new run. The original unrun release remains available with its original evidence and status.

## What we tested

The native cohort contains eight authored conditions across four deterministic layouts: settled completion, missing target, wrong object, wrong position, wrong orientation, unreleased grasp, a moving alias and occlusion. Every fixture has an independent physical reference and a declared expected judgment for each input condition.

Completion requires the correct shoe pose, table support, released robot contact and sufficiently low linear/angular velocity at every recorded sample in a 200 ms native window. The shoe is a rigid, narrow, 30 g proxy. Neither deformable footwear nor real hardware is tested.

Four settled/moving pairs have byte-identical current and goal images and identical complete RGB requests. The moving states have an authored 0.12 m/s velocity. The distinction is deliberately absent from the still image. RGB-only judgment should therefore be `unknown` for both members of each pair. Telemetry supplies presence, pose error, contact measurements and full-window extrema; it is privileged simulator evidence, not a learned camera or contact estimator.

## Results, with the denominators retained

| Measurement | Images only | Images + native telemetry |
| --- | ---: | ---: |
| Valid structured responses | 32/32 | 32/32 |
| Frozen observability-aware verdict matched | 24/32 (75.0%) | 25/32 (78.1%) |
| Known-verdict cases answered correctly | 16/16 | 25/32 |
| Required ambiguous-case abstentions | 8/16 | Not applicable: 0 ambiguous cases |
| Physically incomplete cases called complete | 0/24 | 0/24 |
| Physically complete cases called complete | 0/8 | 1/8 |
| Physically absent targets called present | 0/8 | 0/8 |
| Presence judgment matched its reference | 28/32 | 27/32 |
| Explicit completion/incompletion decisions | 24/32 | 29/32 |
| Physical agreement among explicit decisions | 20/24 | 25/29 |
| Completion-probability Brier score, lower is better | 0.1858 (32 responses) | 0.1174 (32 responses) |

All 24 negative cases per condition also have valid model responses, so planned-negative and valid-negative false-completion denominators are both 24. No service errors or missing responses improve the observed false-completion count. An analytical always-incomplete rule would match 24/32 telemetry reference verdicts, compared with Jev's 25/32. Yet zero false completion alone is insufficient: refusing almost every positive case can produce the same result. The eight physically complete cases per condition must remain beside it.

The two observability-aware accuracies are not a clean improvement estimate. The required answer changes when telemetry makes hidden state observable. These are paired diagnostics across four correlated layouts, not independent samples of a broad robot population. We did not tune the prompt or repeat the cohort after seeing these results.

## Where the judgments fail

**Occlusion defeats completion recognition.** All four occluded fixtures are physically complete. Images-only input requires abstention; telemetry establishes completion. Jev instead answered incomplete in all eight requests. This is a concrete failure of the current frozen evaluation, not evidence that the hidden object was physically absent.

**Telemetry resolves movement but often not successful completion.** Every moving alias was called incomplete with telemetry. In the four visible settled scenes, however, the same condition produced three unknown answers and one complete answer. The remaining four physically complete cases are the occluded scenes, all called incomplete. Our interpretation is that image plausibility, physical evidence and conservative decision rules are not yet reconciled reliably; the output alone does not reveal the model's internal reason.

**Abstention is selective.** Images-only Jev correctly abstained on the four settled and four moving aliases, totaling 8/16 required abstentions. It did not abstain on the four unreleased or four occluded scenes. In the unreleased scenes, an incomplete answer agrees with physical truth but exceeds what the frozen single-image observability contract establishes about contact and dwell.

**No false presence does not mean every absence was identified.** The empty-target scenes received four unknown presence answers with images alone and three absent/one unknown with telemetry. All eight task verdicts were incomplete. The wrong-object scenes supplied the other absent-target cases and were correctly labeled absent in both conditions.

## Measured speed and cost

| Online latency | Images only, 32 attempts | Images + telemetry, 32 attempts |
| --- | ---: | ---: |
| Median | 1.585 s | 1.914 s |
| p95 | 3.548 s | 3.221 s |
| p99 | 4.022 s | 3.339 s |
| Maximum | 4.091 s | 3.371 s |

Across all 64 attempts, median online latency was **1.776 s**, p95 **3.287 s** and maximum **4.091 s**. All attempts succeeded, so all-attempt and successful-response distributions coincide. These are measured hosted-request durations on this run, not actuator reaction times, simulation timesteps or a real-time guarantee.

Timing starts with preloaded PNG bytes and includes request construction, upload, hosted response and parsing. It excludes fixture-file loading, earlier scene rendering, live-camera acquisition, robot motion and physical task duration. Quantiles use the frozen linear interpolation rule. A 64-request cohort cannot establish production tail latency, throughput under concurrency or hardware control performance.

The recorded usage yields an estimated **$0.007644756 input cost** across the 64 requests; every attempt has usage data. This is the frozen client's input-cost estimate, not a billing receipt or a complete service-cost audit. Output-token usage remains in the sanitized run records. The measured model is hosted; this run demonstrates no offline Jev inference or model-weight release.

## What this changes in the architecture

The promising use is a slower supervisory observation layer: assess visible arrangement, identify uncertainty, then request another observation or a bounded physical check. Contact limits, stopping, grasp retention and rapid movement still need local feedback and control. The measured seconds-long round trip should not sit in that fast actuator loop.

For this benchmark, supplied telemetry already contains enough evidence for the frozen deterministic physical rule. A local verifier should evaluate that rule directly and expose its reason. A model can help interpret the image or suggest what to observe next; it should not silently replace an available measured predicate. That is an architectural conclusion from this diagnostic, not a tested new controller.

The next experiment should isolate the failures: compare a deterministic telemetry verifier, the frozen visual evaluator and a separately frozen fusion rule; include occlusion, delayed/incorrect telemetry and an additional camera view. Positive-case recall, false completion, abstention and latency must all remain visible. Any new prompt or decision threshold belongs in a fresh protocol and held-out run.

## Inspect and reproduce

[Current/goal viewer](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/) · [Every recorded request result](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/results.json) · [Frozen evaluation plan](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/evaluation-plan.json) · [Independent run review](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/independent-run-review.json) · [Independent native audit](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/native-validation.json) · [Complete package checksums](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/publication-integrity.json) · [Reproduction and licenses](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-19-image-goal-jev-v02/lab/README.md)

The native generator, client and evaluator remain frozen at [core commit `19acf0734d63ecebc07140c8d84302a54baee4c4`](https://github.com/Dvidia-Inference/dvidia-training/tree/19acf0734d63ecebc07140c8d84302a54baee4c4). The [publication source](https://github.com/Dvidia-Inference/dvidia-training/tree/a3e9afb59f3a9cd7da617cf61416e24fad0a0a5c) adds the measured release and makes positive-case recall explicit in the viewer. The public run retains all 64 rows, typed judgments, probabilities, semantic request hashes, usage and measured timing. Raw provider response bodies and credentials are excluded. The original native, viewer and outer publication inventories remain distinct, and every original fixture and evaluation-plan byte is preserved.

Original DVIDIA code and authored fixture records use MIT. The PiPER asset's upstream MIT notice is retained. Jev remains TypeSafe's hosted model under its own terms. This release contains no trained weights, learned robot policy or physical-robot qualification.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 DVIDIA contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Jev image-goal verification: measured follow-up

DVIDIA robotics research · 19 October 2026.

All 64 frozen Jev requests returned valid structured responses, with no retries or omitted trials. There were zero false-completion declarations among 24 physically incomplete cases per input condition. That result must be read beside poor positive recognition: images alone recognized 0/8 physically complete cases, and images plus privileged simulator telemetry recognized 1/8. All four occluded complete fixtures were called incomplete in both conditions.

Observability-aware verdict agreement was 24/32 for images and 25/32 for images plus telemetry; the conditions have different expected labels, so this difference is not a broad accuracy improvement estimate. The RGB evaluator correctly abstained on 8/16 ambiguous cases. Overall online latency was 1.776 s median, 3.287 s p95 and 4.091 s maximum. These are hosted evaluation times, not robot reaction times.

This publication v0.2 is a measured follow-up to the unchanged v0.1 protocol. The original unrun release remains at `image-goals/v01` on Hugging Face. Frozen native fixtures, prompts, conditions, thresholds and scoring are preserved byte-for-byte. There is no learned policy, trained model-weight release or hardware validation.

Open `index.html` to inspect the current/goal images, positive cases, missing targets and every recorded judgment. `results.json` retains all typed outputs, probabilities, semantic request hashes, usage, timing and independently recomputed scores; raw provider response bodies are omitted. `report.json` is a smaller explicit allowlist for the offline viewer. No credential is included. `evaluation-plan.json` and `cases.json` preserve the original protocol and physical/observability references.

`integrity.json` remains the original native fixture inventory. `report-integrity.json` binds the measured viewer. `publication-integrity.json` binds this full public package. No inventory replaces an earlier seal.

The [independent run review](independent-run-review.json) verifies the frozen requests, append-only attempt ledger, stored provider records and metric arithmetic without rerunning the model.

## Reproduce

Use publication source commit `a3e9afb59f3a9cd7da617cf61416e24fad0a0a5c` (also recorded in `provenance.json`) and install Python 3.12+ with the repository's `.[arm]` extra. The native generator, client and evaluator remain frozen at core commit `19acf0734d63ecebc07140c8d84302a54baee4c4`. PiPER assets are fetched once and verified against the asset lock. Native fixture generation and the static viewer run locally; hosted Jev evaluation requires a TypeSafe account, network access and a securely provided credential.

The frozen evaluation command is `python -m dvidia_training.image_goal_benchmark evaluate --fixtures PATH --output NEW_DIRECTORY`. A new run has new latency and possibly different stochastic outputs; it does not replace this evidence. See `docs/image-goal-benchmark.md` in the source for complete setup and limits. Never paste tokens into a report or source file.

## Independently check the native evidence

`native-validation.json` records the completed independent audit of all 32 physical labels and full-window telemetry extrema. `verify-image-goal-fixtures.py` recomputes those values from the saved native records; it does not call Jev, render images, or run a new physical simulation. Its source hash is recorded in both the audit and publication provenance.

After installing the pinned source above, extract only the original sealed fixture files to a fresh directory. From this downloaded package directory:

```sh
python - <<'PYTHON'
import hashlib, json
from pathlib import Path, PurePosixPath
source = Path.cwd()
destination = source.parent / "image-goal-native-fixtures"
destination.mkdir(exist_ok=False)
inventory = json.loads((source / "integrity.json").read_text())
for item in inventory["files"]:
relative = PurePosixPath(item["path"])
assert not relative.is_absolute() and ".." not in relative.parts
data = (source / relative).read_bytes()
assert len(data) == item["bytes"]
assert hashlib.sha256(data).hexdigest() == item["sha256"]
target = destination / relative
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(data)
for name in ("integrity.json", "evaluation-plan.json"):
(destination / name).write_bytes((source / name).read_bytes())
PYTHON
python verify-image-goal-fixtures.py --fixtures ../image-goal-native-fixtures --output ../native-validation-reproduced.json
```

The verifier intentionally rejects extra files inside the native fixture directory. Keeping the report and its publication metadata outside that directory preserves the original manifest boundary. Expected result: 32 checked cases, 8 physically complete, 24 incomplete, 67 original files verified. Host metadata in a reproduction can differ.

## Licenses and scope

Original DVIDIA code and authored synthetic fixture records: MIT (`LICENSE`). The PiPER simulation asset is from MuJoCo Menagerie commit `0059d4335f8156206f63a35662313385f7ad6d74`; its MIT notice is preserved in `licenses/agilex_piper/LICENSE`. No MuJoCo runtime or upstream mesh files are redistributed in this viewer. DVIDIA marks are brand identifiers; the code license grants no trademark rights.

Jev is TypeSafe's hosted model and retains its own service terms. Its output is evaluated, not used to train or distill a DVIDIA policy. The two model-input conditions are RGB images alone and RGB plus privileged simulator measurements. The latter supplies object identity/presence, pose errors and complete native-window extrema; it is not a demonstrated real-camera or contact estimator. There is no direct motor-control path.

[Research](https://research.dvidia.org/papers/image-goal-jev-v02/) · [Source](https://github.com/Dvidia-Inference/dvidia-training/tree/a3e9afb59f3a9cd7da617cf61416e24fad0a0a5c) · [Hugging Face](https://huggingface.co/spaces/Dvidia/robot-simulation-lab)
Loading
Loading