Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,11 @@ jobs:
dvidia-train --help
dvidia-training-studio --help
dvidia-training-benchmark --help
dvidia-skillspace-coverage --help
dvidia-episode-tape --help
dvidia-training-doctor --json
- name: Test footage, visual training and loopback studio
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor -v
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape -v

arm:
name: Optional arm / Linux / Python 3.12
Expand Down
10 changes: 9 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# DVIDIA Training

[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md)
[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Execution map](docs/roadmap.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md)

[Recorded demo](https://huggingface.co/spaces/Dvidia/dvidia-training) · [Synthetic datasets](https://huggingface.co/datasets/Dvidia/dvidia-training-examples) · [Pilot models](https://huggingface.co/Dvidia/dvidia-training-pilot)

Expand Down Expand Up @@ -32,6 +32,14 @@ Open [http://127.0.0.1:8270](http://127.0.0.1:8270). Select a local Skillspace f

For Windows, use `.venv\Scripts\python.exe` and `.venv\Scripts\dvidia-training-studio.exe`; [complete PowerShell commands](docs/installation.md#windows-powershell) are provided.

## Plan the next useful recording

Open [the collection planner](http://127.0.0.1:8270/coverage) in the same local studio. Define the skill's goal, operating envelope, selected conditions and simulation benchmark. The planner suggests a capture brief for a gap and tallies declared independent reviewed groups. Its example records are fabricated; it does not inspect footage or establish training readiness. Export the plan to keep it.

[Collection design and proposed incentives](docs/skillspace-coverage.md) explain complete attempts, review and rights declarations, source grouping, failures and recoveries. Every skill starts with a benchmark description; an unsupported task environment keeps qualification blocked. Collection targets and measured competence are separate.

The [episode tape recorder](docs/episode-tape.md) preserves the existing simulated arm's measured state, requested/applied commands, contacts and sensor ages. Recording needs the optional arm extra; validation and inspection need only Python's standard library. These tools are available from the current checkout and do not train a new policy or change the older Hugging Face pilot.

## Try the pipeline

Generate an explicitly synthetic example, then train and verify its output:
Expand Down
122 changes: 122 additions & 0 deletions docs/episode-tape.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Simulator episode tapes

Episode tapes preserve what the authored research arm observed, requested and
actually did at each control boundary. They are a separate versioned evidence
format. They do not change the existing movement-training sidecar, train a
supervisor, recover actions from human videos or qualify physical hardware.

The recorder runs locally and offline with the optional `arm` dependencies.
Validation and inspection use only the Python standard library. No renderer,
GPU, FFmpeg, account, API key or network is needed for these operations after
installation. These are dependency requirements; minimum RAM and throughput
have not been established across machines.

## Record fresh evidence

Install the simulator extra using the [installation guide](installation.md).
The commands below run from the checkout with its installed virtual environment.
In Windows PowerShell, replace `.venv/bin/COMMAND` with `.\venv\Scripts\COMMAND.exe`
and put continued commands on one line.
Use a fresh output path outside the frozen benchmark directories:

```sh
mkdir -p runs/episode-tapes
.venv/bin/dvidia-episode-tape record \
--output runs/episode-tapes/nominal-19050.json \
--episode-id nominal-19050 \
--source-recording-id recording-nominal-19050 \
--session-id simulation-session-19050 \
--seed 19050 --case nominal

.venv/bin/dvidia-episode-tape record \
--output runs/episode-tapes/open-jaw-19050.json \
--episode-id open-jaw-19050 \
--source-recording-id recording-open-jaw-19050 \
--session-id simulation-session-19050 \
--seed 19050 --case open-jaw

.venv/bin/dvidia-episode-tape validate runs/episode-tapes/nominal-19050.json
.venv/bin/dvidia-episode-tape inspect runs/episode-tapes/open-jaw-19050.json
```

The `nominal` case records the unchanged authored controller. `open-jaw` keeps
its joint requests while issuing an 80 mm jaw gap. `release-after-lift` requests
an 80 mm jaw gap at control boundaries falling within 0.35 simulation seconds
after the environment first reports a retained lift. With the default 20 ms
control interval, the final such request also governs the remainder of its
step, so the overridden command intervals span 0.36 seconds. The native target
ramp still limits opening speed, and measured aperture may lag the requested
gap. The recorder then returns command authority to the unchanged authored controller. The
interventions change jaw commands and leave native object dynamics intact.
Their outcomes depend on the authored scene and controller; these examples
are recording checks, not a new generalization benchmark.

Related recordings must stay in the same source/session split group. The two
examples above share a session because they reuse the same seeded scene.
Assign stable IDs when collecting evidence; changing an ID cannot make a
duplicate or counterfactual recording independent. The recorder preserves
these IDs but does not assign dataset splits or inspect an existing corpus.

Output creation refuses to overwrite an existing file. A malformed tape is
rejected before a write. A broken native clock or invalid numeric observation
cannot be exported as valid evidence. Inspection never modifies the input.

## What the v1 format preserves

| Component | Meaning |
| --- | --- |
| Provenance | Episode, recording and session IDs; seed and case; authored policy revision; MuJoCo version; scene, configuration and source-file hashes. |
| Clock | Simulation elapsed seconds, with zero after reset settling and a native-clock offset. Wall-clock timing and hardware clock synchronization are outside v1. |
| Initial / before / after | Measured simulator state at zero and at each control boundary. The previous after-state must equal the next before-state. |
| Policy request | Authored joint targets in radians and requested jaw gap in metres. |
| Issued command | The command passed to the environment, separately retaining any declared jaw intervention. Issue time equals the before-state time. |
| Applied command | The step’s start/end times, final ramped targets, final native servo controls and rate-limit flag. These are commands, not measured positions or velocities. |
| Measurements | Joint positions/velocities/actuator torques; jaw actuator forces and measured gap; per-pad normal forces and object contacts; table contacts; tool and object pose/velocity; target position. |
| Authored labels | Action phase and next phase captured after policy evaluation, before the step. These are controller evidence, not approved human labels or deployment inputs. |
| Outcome | Terminal reason, success, validity, simulation duration, step count, authored failure reason and recovery count. These must agree with the final transition. |

Each signal declares its dimensions, units and coordinate frame. Quaternions
use `w, x, y, z`. All recorded signals come from privileged simulator state;
the format does not claim that a physical arm has equivalent sensors. Contact
normal force in newtons is distinct from jaw actuator force. Requested closing
and a measured gap alone are not labels for a successful grasp.

A signal is `available`, `stale` or `missing`. Available and stale readings
retain acquisition time and age. The v1 freshness limit is 0.1 seconds, a
recording-contract choice rather than a validated hardware deadline. Missing
readings have null value, acquisition time and age. They are never fabricated
as zeros. Current native recordings acquire all available signals at the
observation boundary, so their age is zero; stale/missing support is validated
for future data producers but no sensor-fault simulator is added here.

The tape samples at the environment’s control interval, ordinarily 20 ms. It
does not retain the intermediate 1 ms native ticks. Final servo controls
include the environment’s gravity compensation. Applied-command timestamps
describe the entire transition and the final controls, not a claim that the
final command was active throughout that interval. Intra-step force peaks and
short contacts can therefore be missed by this tape.

Validation refuses unsupported versions/scopes, unknown or duplicate fields,
nonfinite/boolean numeric entries, malformed shapes or quaternions, inconsistent
sensor ages, contradictory missingness, reordered or discontinuous observations,
command/transition timing mismatches, unsupported actuator commands, inconsistent
terminal outcomes and changed protocol/configuration hashes. Limits are 10,000
control steps, one million native ticks per control or settling interval, and
64 MiB per tape. The clock origin must match the declared native settling ticks,
and both native jaw controls must equal half the final ramped gap. Hashes
establish consistency, not provenance
authenticity, sensor accuracy or task mastery.

## Next consumers

The first consumers should be a synchronized episode inspector and a frozen
training/confirmation split, followed by a compact state-based outcome model.
The current visual predictor and movement head do not consume these tapes.
A future learner must choose deployable inputs explicitly and hold privileged
object state and authored phase labels out of its deployment feature vector.

Every eventual skill candidate needs its own versioned simulator task,
observable success/failure conditions, fresh held-out scenarios, matched controls,
complete attempted-episode receipts, performance measurements and a supported
robot/gripper envelope. A validated tape is an input to that process; it is not
an installable qualified skill.
8 changes: 8 additions & 0 deletions docs/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The product path is:

```mermaid
flowchart LR
P[Plan skill coverage and simulation benchmark] --> A
A[Skillspace folder, ZIP or manifest link] --> B[Check actual videos and source groups]
B --> C[Fix training, development and test splits]
C --> D[Learn a visual model offline]
Expand Down Expand Up @@ -42,6 +43,13 @@ check the footage, then train and download the result. The server binds only to
your own machine. To use it on your server, keep that binding and forward the
port through SSH. Do not expose this local file-reading interface publicly.

Before collecting, open `/coverage` to define the operating envelope, selected
capture cases and each skill's benchmark description. The [coverage planner](skillspace-coverage.md)
tallies supplied review metadata and exports a companion plan; it does not inspect
videos or run training. See the [execution map](roadmap.md) for the remaining gates.
The [episode tape recorder](episode-tape.md) separately preserves simulated state
and commands for future outcome/phase models; current learners do not consume it.

The default server prohibits downloads. `--allow-online` enables an explicit
per-job option to fetch a public DVIDIA manifest and its declared media. Training
itself remains local. A catalog URL with a claimed demonstration count but no
Expand Down
101 changes: 101 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Execution map: footage to installable skills

Working map, 9 October 2026. This is the team's sequence of builds and evidence
gates. A completed software component does not establish robot competence.

## The product loop

```mermaid
flowchart LR
A[Define skill goal and operating envelope] --> B[Specify its simulation benchmark]
B --> C[Request recordings that fill coverage gaps]
C --> D[Review footage, rights, outcomes and source groups]
D --> E[Freeze development and confirmation groups]
E --> F[Train compatible candidates locally]
F --> G[Evaluate every attempted simulation trial]
G --> H{Meets the declared task criteria?}
H -->|No| I[Diagnose development failures and request new evidence]
I --> C
H -->|Yes| J[Publish versioned candidate, compatibility and results]
J --> K[Install through a compatible adapter and calibrate]
K --> L[Separate physical robot qualification]
```

Confirmation evidence is opened after a candidate is frozen. Once inspected,
that suite becomes regression evidence for later changes. Further qualification
needs fresh confirmation scenarios; it cannot silently reuse a known test set.

## Current foundation and next builds

| Order | Build and owner role | Current state | Evidence needed to advance |
| --- | --- | --- | --- |
| 1 | Collection planning — product + data | Local coverage planner, versioned briefs and declared source-group tallies implemented. | First skill's observable goal, operating envelope, actual recordings and reviewed capture lineage. |
| 2 | Contribution review — data + product | Proposed official-site connection; the local planner imports declarations only. | Immutable review receipts tied to original bytes, plan revision, intended-use permission and complete outcomes; source-group and near-duplicate audits. |
| 3 | Episode evidence — simulation | Recorder preserves state, requests, applied commands, sensor age and authored phases at control boundaries. | Task-specific episode corpus, retained failure cases and synchronized inspector. Tapes are not consumed by a learner yet. |
| 4 | Task and state models — learning | Existing small visual predictor and bounded movement head; outcome/phase supervisor proposed. | Choose deployable sensor inputs, compare with simple baselines and measure held-out boundary, outcome and false-completion errors. Keep authored labels and privileged state outside deployment inputs. |
| 5 | Task simulation — simulation | Existing rigid-box placement adapter and frozen benchmark; other tasks need environments. | Validated assets/contact behavior, robot/gripper/sensor contract, reset distributions and independently observable task outcomes. |
| 6 | Qualification and sample efficiency — evaluation | Local benchmark runner and frozen sample-count evidence exist. | Learning curves over independent source groups, multiple collection orders, frozen candidates, nominal/stress/control strata, all attempted trials and measured cost. |
| 7 | Installable skill — runtime | Simulation candidate capsule and compatibility checks exist. | Bundle model, task, observation/action contract, adapter, data/protocol hashes, calibration requirements and artifact-bound qualification results. |
| 8 | Physical transfer — robotics | Unqualified. | Hardware timing/calibration checks, measured limits and a separate supervised physical evaluation. Simulation results alone cannot advance this stage. |

The coverage planner and recorder are available from the current source checkout.
They are not included in the frozen v0.1.0 release or the older Hugging Face pilot.
No new task policy is trained by adding these components.

## First incoming skills

Start with one bounded task and one supported robot/gripper rather than declaring
a universal skill. For shoe organization, distinguish pairing, alignment and
placing on a shelf: each needs a goal, observable result and task environment.
Record initial conditions, the complete attempt and its outcome. Five- or
six-second clips are useful when they contain that whole transition. Preserve
longer originals and shared recording/session/object IDs across every cut.

Collection starts with selected high-value conditions: different physical
instances, starting poses, views, clutter, failed grasps and recoveries. Initial
quotas are workload estimates. Use development errors to refine the plan;
version any changed brief and re-review earlier evidence against it. Freeze
confirmation groups before fitting or adjusting the policy.

Human footage can support visual representations and task labels. Joint commands,
forces and calibrated geometry require robot telemetry or a validated action
bridge. Synthetic variations preserve their parent lineage and are reported
separately from independently captured evidence. Shoe tying needs a deformable
task environment; it cannot inherit the rigid-box adapter's qualification.

## What “enough data” means

Report independent groups, conditions and outcomes, not just video count or
seconds. Run nested training subsets while keeping development evidence fixed.
Predeclare the target success, false-completion and critical-error limits with
uncertainty estimates and a cost budget. Freeze the selected candidate before
opening fresh confirmation trials. A plateau below the target calls for a
diagnosis of observations, actions, model or simulator, not a “done” badge.

Collection coverage, usable data, trained candidate, simulation qualification and
physical qualification remain separate states. A collection quota cannot promote
any of the later states.

## Every skill's benchmark contract

At creation, record a success predicate, meaningful failure cases and whether an
environment exists. A missing environment blocks qualification while collection
and environment work continue. The planner's description is the start of the
contract; an executable benchmark additionally needs:

- Task/environment, asset, physics, controller and sensor versions; observation
and action units, coordinate frames, degrees of freedom and gripper limits.
- Reset distributions, seeds, episode timeout and nominal, stress, recoverable,
impossible-task and sensor-fault strata where applicable.
- Authored/simple baselines and matched negative controls; a policy hash and
frozen dataset/protocol hashes before evaluation.
- All attempted episodes, including failures, errors and incompatibilities;
success, false completion, recovery and critical-error rates with denominators.
- Wall time, step throughput, end-to-end control latency and memory with named
hardware and measurement boundaries. CPU-only support is measured per task.
- A supported operating envelope and explicit simulation-only or physical scope.

The next engineering milestone is a tape inspector and a small state-based task
outcome model on the existing placement environment. The first real Skillspace
will then test the collection and review loop. See [collection design](skillspace-coverage.md),
[episode evidence](episode-tape.md) and [existing benchmark protocol](benchmarks.md).
Loading
Loading