Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions docs/downloads/reports/Skillspace footage training pipeline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Skillspace footage training pipeline

The public, standalone [DVIDIA training repository](https://github.com/Dvidia-Inference/dvidia-training) provides a local CPU pipeline that turns actual Skillspace media into a fitted visual model and, when aligned numeric actions are supplied, a learned movement head. The useful advance is a reproducible path from source files through provenance checks, grouped splits, measured fitting and portable artifacts. **Video prediction and robot movement remain separate claims.** Ten synthetic six-second clips demonstrate visual fitting; simulator telemetry supplies the movement pilot's action labels. Its exported capsule installs and passes native qualification in one exact scene, while identical joint targets with open jaws fail. Neither experiment learns shoe organization or reconstructs robot actions from human footage. This report documents the implemented path and its limited measured evidence, rather than establishing a sufficient demonstration count or hardware competence. ([Standalone source archive](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/release/dvidia-training-v0.1.zip), [Qualification receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/qualification/qualification.json))

## Actual media and connected source groups gate training

Start with the repository's [installation guide](https://github.com/Dvidia-Inference/dvidia-training/blob/main/docs/installation.md) and [hardware guide](https://github.com/Dvidia-Inference/dvidia-training/blob/main/docs/hardware.md). The pipeline accepts a local folder, ZIP or prepared dataset; explicitly enabled online mode also accepts a supported public DVIDIA manifest and its declared media. Visual training requires Python 3.12+, NumPy and FFmpeg/ffprobe; arm simulation adds MuJoCo. A Skillspace description with no accessible video does not become a dataset. Images alone are rejected. An audit-only preparation step checks media, copies the exact bytes, records hashes and freezes train, development and test assignments before fitting. Fresh output directories prevent silent replacement of earlier runs. The command-line interface and a loopback training studio expose this same path. ([Implementation](https://github.com/Dvidia-Inference/dvidia-training))

Split integrity depends on **connected source groups**, rather than independent filenames. Clips sharing a source recording, session, shoe-pair identifier or identical media bytes stay together. At least three groups are required to populate all three splits. Plain video folders can enter with explicit lineage warnings, but their per-file groups do not prove that recordings are independent. A supplied “complete” flag records the contributor's declaration; it does not establish that the entire task appears or that the demonstration is correct. Thus the implementation prevents declared lineage from crossing splits while retaining the unresolved need for semantic and provenance review. ([Example visual Skillspace](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/visual-skillspace.zip))

## Ten synthetic clips establish visual fitting, not a shoe skill

The visual pilot uses ten encoded six-second moving-disc clips, with seven training, two development and one test recording. Each contributes twelve sampled frames. The learner projects low-resolution RGB through a basis fitted only on training data, then fits an affine ridge transition between successive representations. Development error selects regularization; the untouched test recording measures prediction against persistence and a training-mean baseline. This compact CPU learner checks the training path without requiring a large pretrained vision model. ([Visual measurements](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/visual-training/training/report.json))

Three standalone repeats on a macOS Apple M5 host with 24 GiB RAM take 0.5315, 0.5106 and 0.5102 seconds for intake, fitting and export: **median 0.5106 seconds**. Dependency installation, imports and fixture creation are excluded; integrity inspection is timed separately. The Python process lifetime peak RSS is **48.13 MiB**, excluding FFmpeg/ffprobe children. Repeats reuse the same test recording and establish repeatability, not additional success evidence. These measurements describe this small default learner; they do not establish minimum hardware or scaling to larger video models. ([CPU benchmark receipt](https://github.com/Dvidia-Inference/dvidia-training/blob/main/benchmarks/macos-m5.json))

| Test measurement | Learned model | Baseline | Evidence unit |
|---|---:|---:|---|
| Normalized RGB mean squared error | 0.0003039 | 0.0017714, persistence | One video, eleven transitions |
| Joint-delta mean squared error, rad² | 5.768 × 10⁻⁷ | 8.173 × 10⁻⁵, zero delta | One telemetry recording, 621 samples |

The visual result supports a narrow conclusion: the fitted transition predicts these synthetic image samples better than copying the preceding frame. Eleven correlated transitions are not eleven independent task trials. Pixel error measures neither grasping nor placement success, and the model observes no calibrated depth, contact force or human action. **No shoe demonstration was used.** The exported visual artifact contains numeric weights, a model contract, measurements and a dataset receipt; original media remain in the local prepared dataset. An integrity inspection checks the exported files and ZIP hashes. ([Visual measurements](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/visual-training/training/report.json), [Training implementation](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/release/dvidia-training-v0.1.zip))

## Executable movement requires aligned actions and an adapter

Movement fitting is conditional on action sidecars with timestamped context, target error and six joint deltas in the existing feature contract. Each sidecar binds its media and samples by hash and declares an alignment method and supporting evidence. Accepted origins include teleoperation, a validated video-action bridge and native simulator telemetry. These fields check internal consistency; a declaration does not independently validate action recovery from pixels. The standalone pilot uses the third origin: ten separately executed native simulator rollouts with state and authored targets, accompanied by schematic footage. ([Native telemetry Skillspace](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/native-skillspace.zip))

Its grouped split contains seven training, two development and one test recording, totaling 116.5 seconds of footage. A normalized radial-basis ridge model fits **4,029 training labels**, chooses regularization on development data and reaches the joint-delta error shown above on 621 test samples. This is an offline prediction result from one held-out recording. It does not measure closed-loop success, independent scene generalization or improvement over the earlier controller. The learned head maps privileged simulator context and a requested goal to joint deltas; phases, waypoints, grasping and recovery remain authored. This scope follows the earlier skill-capsule prototype. ([Movement measurements](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/native-training/movement/report.json), [Run receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/native-training/run.json), [Skill capsule acquisition](https://research.dvidia.org/papers/skill-capsule-acquisition/))

An explicit supported task source permits export of a runtime-bound simulation capsule. The standalone run records **1.482 seconds on its CPU pipeline timer**, excluding final ZIP creation and integrity verification. It exports a 14,843-byte candidate, passes inspection and installs it. This describes one small run, not a throughput comparison. An earlier filename-handoff failure remains recorded in the release ledger. ([Run receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/native-training/run.json), [Release ledger](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/release-ledger.json))

Installation alone does not enable qualified execution. A separate native test in the exact scene of the held-out recording completes placement in **12.952 simulated seconds with 0.311 mm target error**. Replaying the same joint targets with open jaws fails at the 18-second horizon. This supports the contact-dependent execution path in that scene; it adds no independent scene sample or hardware evidence. Artifact hashes establish which candidate and runtime were tested, not physical accuracy. ([Qualification receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/qualification/qualification.json), [Native replay](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/qualification/replay))

## The next experiment should measure the missing bridge

The current local path avoids API dependence once its tools and data are installed. Local media decoding restricts FFmpeg protocols, while optional remote intake is explicit. Separate continuous-integration checks passed visual fitting and studio tests on Linux, macOS and Windows with Python 3.12 and 3.13, the optional Linux arm pilot, and an actual Docker example, training run and inspection under `--network none`. The image build required network access; this does not establish installation on a new air-gapped machine or minimum hardware. No GPU scaling, throughput improvement over another implementation or hardware transfer follows from the pilots. The package makes experiments inspectable and repeatable; its scientific value depends on what demonstrations, labels and task environments enter it. ([Passing software checks](https://github.com/Dvidia-Inference/dvidia-training/actions/runs/37740260434), [Source and training guide](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-08-footage-pipeline-v1/release/dvidia-training-v0.1.zip))

For shoe organization, the next decisive step is a reviewed corpus of complete one-shoe-to-marked-slot attempts and a validated observation-to-action bridge for a defined arm, gripper and sensing arrangement. Sample-count comparisons should hold architecture, task definition and evaluation fixed, group related originals, and report independent closed-loop episodes alongside prediction error. Until those components exist, a shorter visual training run cannot answer how many human videos make a robot competent. The pipeline's contribution is to preserve those distinctions in executable product gates, making the missing evidence measurable before a candidate becomes a downloadable skill.
8 changes: 8 additions & 0 deletions docs/downloads/topics/physical-grounding/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Physical grounding

[Skillspace footage training pipeline](../../reports/Skillspace%20footage%20training%20pipeline.md)
introduces the separate public [DVIDIA Training](https://github.com/Dvidia-Inference/dvidia-training)
repository: CPU footage intake, connected source groups, measured visual fitting,
optional aligned movement training and simulation capsule export. Hardware guidance,
installation tools and synthetic examples are included. One exact native scene passes
with a failed open-jaw control; human-video action recovery and shoe competence remain
unestablished. [Release and evidence](runs/2026-10-08-footage-pipeline-v1/README.md).

## Installable learned movement capsule — 7 October 2026

[Skill capsule acquisition and optimization](../../reports/Skill%20capsule%20acquisition%20and%20optimization.md) packages a learned CPU movement head inside an authored placement supervisor. A new frozen evaluation preserves teacher/student 7/8 outcomes, with no overall speedup. The local lab imports the JSON capsule, qualifies the exact scene with native placement and an identical-command/open-jaw control, and runs without retraining. Changed inputs require another test. [Source and evidence](runs/2026-10-07-skill-capsule-v1/README.md), [capsule](runs/2026-10-07-skill-capsule-v1/simlab/placement_candidate.skill.json), [browser-run replay](runs/2026-10-07-skill-capsule-v1/demo/replay.html). The capsule does not supply video learning or physical qualification.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Skillspace footage training tools

The training product has its own public, MIT-licensed repository:
[DVIDIA Training](https://github.com/Dvidia-Inference/dvidia-training).
The [v0.1.0 alpha release](https://github.com/Dvidia-Inference/dvidia-training/releases/tag/v0.1.0)
includes the installer, source, examples, result archives and checksums.
Start with its [installation guide](https://github.com/Dvidia-Inference/dvidia-training/blob/main/docs/installation.md),
[hardware guidance](https://github.com/Dvidia-Inference/dvidia-training/blob/main/docs/hardware.md)
and [Skillspace format](https://github.com/Dvidia-Inference/dvidia-training/blob/main/docs/skillspace-format.md).

The base package uses NumPy and system FFmpeg/ffprobe on a CPU. MuJoCo is an
optional extra for movement candidates and native simulation. The local studio
provides footage checks, grouped splits, offline fitting and a verified download.
It never turns a visual score into a robot task-success score.

[Download source](release/dvidia-training-v0.1.zip) ·
[Python wheel](release/dvidia_training-0.1.0-py3-none-any.whl) ·
[Visual example Skillspace](visual-skillspace.zip) ·
[Native telemetry example Skillspace](native-skillspace.zip)

The ten visual example videos each last six seconds. They are authored synthetic
moving-disc recordings, with seven training, two development and one test group.
The first learner fits train-only PCA and a ridge next-frame predictor. Held-out
RGB error is 0.000303879 against 0.001771414 persistence error, measured on one
recording and eleven correlated transitions. The [CPU benchmark](cpu-benchmark.json)
records a median 0.5106-second full pipeline run on an Apple M5 host and a
50,462,720-byte Python-process lifetime peak RSS, excluding child decoders.
These measurements do not establish minimum hardware requirements.

The optional native example has ten recorded simulation placement attempts.
Aligned state and authored target labels train the movement head on 4,029
samples, with 1,154 development and 621 test samples from separate recordings.
The [movement report](native-training/movement/report.json) records offline
joint-delta error. The [candidate capsule](native-training/candidate.skill-capsule.json)
is bound to the source archive's exact runtime and still uses authored gripper,
contact, phase and recovery logic plus privileged simulator state.

Its [local qualification](qualification/qualification.json) passed one exact
held-out synthetic scene. An identical arm-target sequence with open jaws failed.
[View the measured replay](qualification/replay.html). This establishes a local
simulation test of this candidate; it supplies no human-video action bridge,
shoe-organizing skill, general robot competence or physical robot qualification.

The [release ledger](release-ledger.json) records scope and the first integration
failure. [Visual result](visual-training/training-result.zip) and
[movement result](native-training/training-result.zip) downloads contain model
data and receipts, excluding original footage. The example Skillspace archives
separately contain the authored recordings and, for the native example, source
telemetry evidence. [Manifest](manifest.json) hashes every published artifact.

For reproduction, install the package and tools, extract one example Skillspace,
and run `dvidia-train run /path/to/extracted-skillspace --output runs/new-run`.
The native branch requires `.[arm]` and
`--task-source /path/to/extracted-skillspace/task.skill.json` to export a capsule.
Use a fresh output directory. See the separate repository's
[native pilot script](https://github.com/Dvidia-Inference/dvidia-training/blob/main/benchmarks/native_pilot.py)
for the complete collect, train, install and local qualification sequence.
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
{
"schema_version": 1,
"kind": "dvidia.small-cpu-pipeline-benchmark",
"scope": "synthetic-software-throughput",
"gpu_used": false,
"hardware": {
"system": "Darwin",
"architecture": "arm64",
"python": "3.12.9",
"cpu_model": "Apple M5",
"logical_cpu_count": 10,
"physical_ram_bytes": 25769803776,
"host_specification_method": "Local system queries; capacity is not minimum required memory."
},
"dependencies": {
"numpy": "2.5.3",
"ffmpeg": "ffmpeg version 9.0.1 Copyright (c) 2000-2026 the FFmpeg developers"
},
"synthetic_source_creation_seconds": 0.46001008385792375,
"python_process_lifetime_peak_rss_bytes": 50462720,
"trials": [
{
"repeat": 0,
"pipeline_wall_seconds": 0.5314955001231283,
"pipeline_report_seconds": 0.5286902498919517,
"integrity_inspection_seconds": 0.0021797919180244207,
"test_rgb_mse": 0.00030387928412713475,
"persistence_rgb_mse": 0.0017714138375595212,
"counts": {
"train": 7,
"dev": 2,
"test": 1,
"episodes": 10,
"groups": 10,
"total_seconds": 60.0,
"declared_complete": 10
},
"gpu_required": false
},
{
"repeat": 1,
"pipeline_wall_seconds": 0.5106192079838365,
"pipeline_report_seconds": 0.5084841668140143,
"integrity_inspection_seconds": 0.0017631668597459793,
"test_rgb_mse": 0.00030387928412713475,
"persistence_rgb_mse": 0.0017714138375595212,
"counts": {
"train": 7,
"dev": 2,
"test": 1,
"episodes": 10,
"groups": 10,
"total_seconds": 60.0,
"declared_complete": 10
},
"gpu_required": false
},
{
"repeat": 2,
"pipeline_wall_seconds": 0.5101718751247972,
"pipeline_report_seconds": 0.5080305421724916,
"integrity_inspection_seconds": 0.0017214161343872547,
"test_rgb_mse": 0.00030387928412713475,
"persistence_rgb_mse": 0.0017714138375595212,
"counts": {
"train": 7,
"dev": 2,
"test": 1,
"episodes": 10,
"groups": 10,
"total_seconds": 60.0,
"declared_complete": 10
},
"gpu_required": false
}
],
"limits": [
"Python RSS excludes FFmpeg/ffprobe child processes and is not total memory or a minimum requirement.",
"One machine, default tiny-resolution learner, ten synthetic clips; no hardware scaling claim.",
"The same held-out recording is re-scored for repeatability, not independent success evidence.",
"Wall time includes intake, fit and export; inspection is timed separately; dependency installation/imports are excluded."
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"object_position": [0.34, 0.07, 0.31], "target_position": [0.44, -0.006, 0.31], "object_size": [0.04, 0.04, 0.04], "object_mass": 0.04, "object_friction": 0.8, "jaw_force_limit": 15.0, "joint_torque_limit": 40.0, "seed": 8006}
Loading
Loading