Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ All four covers are **AI-generated editorial illustrations**, not footage, measu

## Latest study

[Clef and grounded video observations](reports/Clef%20and%20grounded%20video%20observations.md): five licensed POV excerpts processed, with actual costs, timings and important caption/checker failures. This is an engineering feasibility result; the separate 40-episode accuracy pilot still needs its reference set.

[From Skillspace to robot skill](reports/From%20Skillspace%20to%20robot%20skill.md) reviews Argus and proposes a measurable path from accepted videos to robot-tested releases. Includes a current implementation gap analysis, DOF compatibility requirements and an unrun [annotation pilot](topics/evaluation-provenance/experiments/0002-skill-graduation.md).

## A useful first contribution
Expand Down
12 changes: 11 additions & 1 deletion docs/topics.json
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@
"title": "Evaluation and provenance",
"kicker": "Reproducible evidence",
"summary": "Test episode readback, original-to-derivative lineage and split integrity using small, auditable fixtures.",
"status": "scoping",
"status": "evaluating",
"cover": "assets/evaluation-provenance.webp",
"imageAlt": "AI editorial illustration of a notebook with contact-sheet photos; not a completed validation result.",
"updated": "2026-10-06",
Expand Down Expand Up @@ -250,6 +250,12 @@
"text": "Do grounded temporal annotations reduce false completion labels, and what evidence should gate a robot-tested skill release?",
"state": "open",
"issueUrl": "https://github.com/Dvidia-Inference/dvidia-research/issues/5"
},
{
"id": "ep-05",
"text": "Can visual checks detect caption mistakes and missing state changes on a held-out human-referenced set?",
"state": "open",
"issueUrl": "https://github.com/Dvidia-Inference/dvidia-research/issues/5"
}
],
"milestones": [
Expand All @@ -268,6 +274,10 @@
{
"label": "Independent result review",
"done": false
},
{
"label": "Five-clip image API feasibility run published (not accuracy)",
"done": true
}
],
"firstTask": {
Expand Down
84 changes: 84 additions & 0 deletions reports/Clef and grounded video observations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Clef and grounded video observations

**Engineering pilot executed · 6 October 2026 · Accuracy evaluation outstanding**

We executed a small, preregistered feasibility run: five existing public, licensed three-second POV excerpts, four original-timestamped frames per excerpt, one Gemma caption call and one Clef-flash check per excerpt. All five completed. The estimated inference cost was **$0.00140634**. This establishes that the image APIs and bounded data contract work; it does not establish that the annotations are accurate.

The useful finding was a failure: a caption called a pink plate a cutting board, and the second model agreed. Several descriptions listed objects or hand positions while missing visible folding or transfer. **Model agreement must not become a “verified” badge.** The observations below come from AI-assisted inspection of the exact input frames after outputs were frozen, not independent human annotations.

[Preregistration](../topics/evaluation-provenance/runs/2026-10-06-clef-feasibility/preregistration.md) · [Complete sanitized results and source/frame hashes](../topics/evaluation-provenance/runs/2026-10-06-clef-feasibility/results.json) · [Prompts and request recipes](../topics/evaluation-provenance/runs/2026-10-06-clef-feasibility/request-recipes.json) · [Next evaluation protocol](../topics/evaluation-provenance/experiments/0002-skill-graduation.md)

The request recipes were reconstructed after execution by replaying the frozen responses through the unchanged adapter, with all five normalized results matching. They retain prompts, schemas, model parameters and frame hashes in place of image bytes. They are a reproducibility aid, not a captured wire log.

## Is Clef a better JEV?

They share the useful pattern of state plus typed questions producing decisions. Our current JEV integration still provides category-based tag suggestions. Jev 1.13 accepts text, not images or video, and does not generate descriptions. Its documented price is $0.042 per million input tokens. [TypeSafe model reference](https://docs.typesafe.ai/models), [typed primitives](https://docs.typesafe.ai/introduction).

Cloudflare presents Clef as an open, JEV-compatible decision model with visual input. That makes it a candidate for judgments grounded in actual frames rather than only a coarse category. The announcement's comparisons are vendor evaluations, not an evaluation of DVIDIA footage. [Cloudflare announcement](https://blog.cloudflare.com/clef-decision-models/).

| Role | Candidate used or retained | Boundary |
| --- | --- | --- |
| Broad text classification / routing | Existing JEV 1.13 path | Cannot inspect pixels; a label derived from text is not a visual check |
| Readable observation | Gemma 4 26B A4B | Generates a description; can miss changes or invent details |
| Bounded visual judgment | Clef-flash | Directly checks sampled images; can still agree with an incorrect caption |
| Hand/object coordinates | Separate measured tracker, not implemented by this pilot | Captions and decision scores supply neither meshes nor boxes |
| Robot control | Separate policy and robot profile | No control actions or robot learning demonstrated here |

For the hosted Clef APIs, use the actual input schema: at most four embedded images per request, not an arbitrary video URL. Clef and Clef-flash are listed at $0.24 and $0.09 per million input tokens respectively. Local model-card video support must not be confused with the hosted image endpoint. [Clef API](https://developers.cloudflare.com/workers-ai/models/clef/), [Clef-flash API](https://developers.cloudflare.com/workers-ai/models/clef-flash/), [model card](https://huggingface.co/Cloudflare/clef).

Gemma supplies the generated text in this experiment; its listed rates are $0.10 per million input tokens and $0.30 per million output tokens. No conclusion about it being the best captioner follows from this run. [Gemma API](https://developers.cloudflare.com/workers-ai/models/gemma-4-26b-a4b-it/).

**Decision:** retain JEV for its existing text decisions and evaluate Clef as an additional visual check. There was no head-to-head labeled test, so we cannot rank their accuracy.

The requested Grok CLI research was attempted twice, but the runs ended before producing a sourced memo. A direct X search for “Clef Cloudflare” reached a sign-in wall; indexed X search supplied no reliable result. We therefore attribute no community consensus or X-specific finding. The technical comparison above uses primary documentation and the actual API run.

## What ran

Fixed inputs were the local `onion`, `melon`, `shirt`, `jeans` and `pack` excerpts from EgoAnnotate, attributed to Taher Panbiharwala and Zainab Barwaniwala under CC BY 4.0. The result manifest links each source at dataset revision `156d13adaee4110cf63e54fa940ec16ff199d12f`. Hashes identify the exact excerpt bytes; full upstream originals were not independently hash-verified. No private contributor footage was used.

Each input supplied four ordered frames at 0, 1, 2 and approximately 2.95833 seconds, resized to a maximum 512-pixel side. The models received the images and timing, not the filenames or publisher task labels. No prompt changes or automatic retries occurred after inspecting results.

| Measurement | Observed result |
| --- | --- |
| Windows with schema-valid results | 5 / 5 |
| Image inference calls | 10 |
| Provider-reported input / output tokens | 12,682 / 665 |
| Price-derived estimate | $0.00140634; not an invoice or total operating cost |
| Gemma latency, median / range | 5.378 s / 3.762–29.025 s |
| Clef-flash latency, median / range | 1.704 s / 1.448–2.825 s |
| Clef support decisions | 5 “supported”; no accuracy implication |
| Accepted local outcome | 5 “unknown” |

An earlier synthetic text-only schema probe is separately recorded and excluded from these image-run totals. Extraction, storage, networking, review, taxes and application operations are not included in the estimate. Five tiny excerpts cannot estimate production latency or cost distributions.

| Excerpt | Post-run inspection finding |
| --- | --- |
| Onion | Pink plate apparently mislabeled as cutting board; checker agreed |
| Melon | Visible knife/watermelon identified; cutting progression underdescribed |
| Shirt | Inward fold visible in final frame; caption mostly described hands |
| Jeans | Folded shape visible across samples; caption mostly described holding |
| Pack | Item moves toward a container; transfer underdescribed |

These are qualitative assistant observations, not five independently adjudicated correctness labels. All raw responses and the inspection method remain in the results. Unknown local endings are preferable to fabricated task completion, but abstention alone does not establish a useful recognizer.

## Product integration and next research

The implementation path is a separately versioned annotation alongside existing capture checks. It binds source hash/bytes, original PTS, frame digests, sampling windows, model identifiers and recipe. Generated observations stay distinct from creator notes. The interface should show a small timeline, tap-to-seek moments and optional tag/description acceptance. Agreement reads “Models agree,” with the overall result explicitly unreviewed. One video decoder is sufficient.

Extraction runs next to the original. Up to six windows and 24 stills are a bounded first implementation, with gaps explicitly shown. New frame transfer requires its own opt-in; old coarse-tag consent does not grant it. Model failures never block saving. Durable attempt leases prevent refreshes from creating repeated billable calls; deletion/expiry remove derived evidence. This public report describes the contract, not private implementation code. Live activation and deployment evidence are separate from this engineering experiment.

The original [40-episode pilot](../topics/evaluation-provenance/experiments/0002-skill-graduation.md) remains **unrun**. This five-clip feasibility check is not a replacement for its grouped development/test split, human references or success criteria. Next steps:

- [ ] Collect the consented one-task cohort, including incomplete attempts and recoveries; freeze session-grouped splits.
- [ ] Establish independent reference labels for objects, state changes, local outcomes and temporal intervals.
- [ ] Improve the caption recipe on development data: explicitly compare before/after state and identify the manipulated object and relation.
- [ ] Compare caption-only, caption-plus-Clef and the existing coarse baseline under fixed input and cost budgets; count checker-agreed errors.
- [ ] Evaluate a stronger temporal visual model and denser sampling separately; report the cost/recall tradeoff on untouched test data.
- [ ] Persist accepted annotation revisions with account originals and task versions; retain model and creator provenance.
- [ ] Add tracking only with actual coordinates, timing, visibility and uncertainty; no decorative meshes presented as measurements.

## From a command to a learned physical skill

“Put both bottles in the cart” needs an observable goal, objects and starting conditions. A useful annotation might describe reaching, grasping, lifting, transporting and releasing, each with evidence and uncertainty. These are the proposed representation, not observations from our five clips or the user's screenshot.

That symbolic sequence is not muscle memory. The latter analogy fits a learned control policy only after it maps observations to actions on a specified robot, with measured feedback and evaluation. DOF counts alone do not define compatibility. Keep the chain explicit: **observations → accepted task episodes → versioned dataset → robot-specific training → held-out physical evaluation**. A Skillspace can organize that history, but more videos do not automatically advance its readiness. See [the graduation and compatibility proposal](From%20Skillspace%20to%20robot%20skill.md).
4 changes: 3 additions & 1 deletion reports/From Skillspace to robot skill.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# From Skillspace to robot skill

**Source review and proposal · 6 October 2026 · Experiments not run**
**Source review and proposal · 6 October 2026 · Graduation experiment not run**

Follow-up: the [Clef image feasibility pilot](Clef%20and%20grounded%20video%20observations.md) has now run on five public excerpts. It does not complete the 40-episode accuracy protocol below.

A Skillspace can become the home for a skill's learning history. More uploaded videos do not, by themselves, establish a better policy. We need evidence that new demonstrations add useful coverage, survive quality checks, and improve held-out performance on a specified robot.

Expand Down
4 changes: 2 additions & 2 deletions topics/evaluation-provenance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

**Question:** Test episode readback, original-to-derivative lineage and split integrity using small, auditable fixtures.

Status: **scoping** · Updated 6 October 2026 · Results: **not run**
Status: **evaluating** · Updated 6 October 2026 · Results: **image feasibility run complete; reference accuracy and provenance baselines not run**

[Open a research question](https://github.com/Dvidia-Inference/dvidia-research/issues/new?template=research-question.yml) · [Baseline protocol](experiments/0001-baseline.md) · [Bibliography](references.bib)

Expand Down Expand Up @@ -48,7 +48,7 @@ All sample sizes, thresholds and resource ceilings here are proposed. See the [f

## Results

**Not run.** Source review is complete; no experiment, validation claim or independent reproduction is complete. Add measured results and their limitations only after execution.
The [five-clip Clef feasibility run](../../reports/Clef%20and%20grounded%20video%20observations.md) completed ten image calls for an estimated $0.00140634. It exposed caption omissions and a checker-agreed object error. [Raw results and preregistration](runs/2026-10-06-clef-feasibility/) are available; independent review and the original baselines remain outstanding.

[ramos2021rlds]: https://arxiv.org/abs/2111.02767
[rlds2026schema]: https://github.com/google-research/rlds
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Preregistered hosted temporal-vision engineering pilot

Registered before model calls or frame inspection. Date: 2026-10-06.

Fixed sample: five existing public/ego/sources/{onion,melon,shirt,jeans,pack}.mp4 files in that order. No replacement based on results. Sources are existing public EgoAnnotate CC BY 4.0 examples with attribution in public/ego/sources/provenance.json; the pilot makes no claim of original full-dataset hash verification. Frame timestamps are relative to these exact local source bytes.

Protocol: source-frames-v1 extractor selects at most four presentation-timestamp frames in each bounded <=6 second window. For each clip, select only its first and last available windows, deduplicating a single-window clip. This is at most 10 windows, each receives exactly one Gemma caption request followed by one independent image-grounded Clef check. No automatic retries or outcome-based reselection. Maximum 20 provider calls, total engineering ceiling $1. Reserve $0.03 for each attempt; stop before exceeding $0.60 reserved (20 attempts) or $1 reported estimated spend. The reservation exceeds documented per-call worst case for Gemma's 256k context at $0.10/M plus bounded 768 output tokens at $0.30/M; Clef context 65,536 at $0.09/M. Endpoint only official api.cloudflare.com. Errors/latencies/unknowns retained. Missing usage is reported as missing/failure, never zero inferred.

Gemma receives only sampled images and exact source timestamps, no filename, topic title, publisher task labels or reference outcomes. It proposes a plain factual description, visible objects, short action tags, sampled action outcome, and final-frame evidence. Clef receives the same ordered images and treats the proposed caption as an untrusted claim, independently checking support and whether the last frame visibly supports success/failure/unknown. Exposed outcome is unknown unless independent support and clear final evidence agree. No text embedded in frames may direct either model.

Evidence logged: immutable source hash/bytes, actual selected frame IDs/PTS/seconds/hashes, window bounds, model IDs, sanitized raw responses, measured latency, provider token usage, price-derived estimated cost, final bounded outputs/errors. No keys, account IDs, private media or source code in results. Images remain local derived evidence only.

Feasibility criteria: requests accepted with real images; bounded schema parses; time/usage recorded; original frame provenance preserved; no fabricated completion or calibrated confidence. No benchmark accuracy claim. Human spot review occurs only after outputs are frozen and reports concerns, not ground-truth success percentages. Selection is small and nonrandom; sparse images miss between-frame motion, audio and full task completion; direct captions and agreement can both be wrong. Existing JEV analysis remains separate.

Fixed files:

[
{
"id": "onion",
"path": "public/ego/sources/onion.mp4",
"bytes": 209273,
"sha256": "100e25698ee99e99dc648c41d9864144e312f0fce7c07dc9a7b06b487b9a2a5f"
},
{
"id": "melon",
"path": "public/ego/sources/melon.mp4",
"bytes": 298547,
"sha256": "f32cf75d28602d169b4f302830ba633a351ac890596887c54c8e928efcf2650e"
},
{
"id": "shirt",
"path": "public/ego/sources/shirt.mp4",
"bytes": 468174,
"sha256": "8392c62f4e9590a15745311746114c2b6355e24c60fa87fef68e834f20762113"
},
{
"id": "jeans",
"path": "public/ego/sources/jeans.mp4",
"bytes": 538358,
"sha256": "95845362951b3f78b7b731691df6d472bf176ced4b07ecef4d99d0b8a21da3e2"
},
{
"id": "pack",
"path": "public/ego/sources/pack.mp4",
"bytes": 394403,
"sha256": "87a67e5b10f124a65df7d669086d4a9b2cb7ae25a5d758594b9b3aa7b0938f4c"
}
]
Loading
Loading