Skip to content

SharpaDex 1.0 ingestion and training - #742

Open
jaynye wants to merge 23 commits into
mainfrom
jmni/sharpa-bringup
Open

SharpaDex 1.0 ingestion and training#742
jaynye wants to merge 23 commits into
mainfrom
jmni/sharpa-bringup

Conversation

@jaynye

@jaynye jaynye commented Oct 25, 2026

Copy link
Copy Markdown

Store, validate, and train on SharpaDex-1.0. Adds

  1. dexmate_bimanual embodiment as ID 8 with arm, hand, keypoint, tactile, head-link validation (missing arrays, incorrect shapes, non-finite values, limit violations). Camera intrinsics not provided with dataset so added a permission rule to bypass requirement for Sharpa Dexmate
  2. LeRobot-to-Zarr converter, which computes kinematics using URDFs and performs consistency and verification checks. Added use details and updates to the data-contribution documentation and validated on 115 episodes
  3. HPT keypoints training recipe which reuses human keypoints transform pipeline and processes keypoints in robot's head-link frame (140d) and is validated for 5 training steps

IMPORTANT!!! Existing human and EVA datasets need no reprocessing, but existing recipes will recompute normalization statistics once due to loader changes

jaynye and others added 23 commits October 24, 2026 16:05
@jaynye @claude
rldb: add dexmate_bimanual embodiment and its validation interface
165c67e 
Interface for storing SharpaDex episodes (a Dexmate robot with two Sharpa
Wave hands):

- EMBODIMENT.DEXMATE_BIMANUAL = 8 and "dexmate_bimanual" in
  episode_rules.EMBODIMENTS. 7 stays free: unmerged branches use it for
  YAM_BIMANUAL. Existing ids are unchanged.
- MCAP keeps its own embodiment subset (human_*, eva_*), so an MCAP episode
  declaring dexmate_bimanual is rejected instead of being checked against
  the gripper ROBOT profile.
- rldb/zarr/dexmate_rules.py (torch-free): key names, widths, URDF hand
  joint limits and margins, and check_dexmate(store, T, errors, warnings),
  which the validator runs for dexmate_bimanual only.
- Intrinsics: ZarrWriter accepts intrinsics=MissingMetadata() only for
  dexmate_bimanual and stores an explicit null. The validator errors on an
  absent key for every embodiment; an explicit null is an error, or a
  warning with --allow-missing, for Dexmate, and stays an error elsewhere.
  camera_errors gains a keyword-only allow_null_intrinsics=False, so every
  existing caller is unchanged.
- package-data for the converter's kinematics assets
  (egomimic/scripts/sharpa_process/kinematics/).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
zarr: test the dexmate rules on padded episodes and at the margins
ba77ecc 
Padding rows past total_frames are never read; hand-limit errors and
warnings switch exactly at ERROR_MARGIN_RAD and WARN_MARGIN_RAD (commands
only warn); optional cmd_hand_keypoints are shape- and NaN-checked when
present; one-frame spikes and values beyond 2*pi warn with the joint name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
rldb: Dexmate embodiment, keypoints in the head link frame
697ee31 
Dexmate reuses Human's keypoints keymap and transform list: it reads
{side}.obs_hand_keypoints and {side}.obs_ee_pose (palm) into the Human
batch names and re-expresses them in obs_head_link_pose, which works
because every track is in the robot base frame. actions_keypoints is 140
wide with quaternion palms (138 YPR). Wrist cameras are optional
(wrist_cameras=False); there is no Pi mode.

Tested on a padded synthetic episode with a moving head: the whole chunk
is in the current head link frame, and past the last frame it repeats
that frame instead of reading the zero padding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
configs: SharpaDex HPT keypoints recipe for dexmate_bimanual
f969eb7 
- data/dexmate_keypoints.yaml: Dexmate keypoints keymap (wrist cameras on)
  and keypoints_headframe_quat, stride 1, lab sharpa_dex.
- model/hpt_bc_keypoints_dexmate.yaml: dims 140/140, domain and action key,
  state_keypoints and wrist stems, front/wrist encoders, FM head.
- train_zarr_dexmate_keypoints.yaml: both, without the video evaluator; the
  episodes have no camera calibration to draw keypoints with.
- test_model_dims EXPECTED row; the valid split follows train.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
tests: Dexmate HPT train step and checkpoint save/reload
bc3ebf7 
The (dexmate, hpt) recipe trains 2 CPU steps on synthetic dexmate_bimanual
episodes (null intrinsics, wrist cameras) through trainHydra.train(), and
the batch carries actions_keypoints, the keypoint state and all three
cameras. The shared overrides disable checkpointing, so a second test
saves a checkpoint, rebuilds the model from it alone (norm stats keyed by
id 8, identical weights) and resumes training from it. Dexmate stays out of
the Pi parametrization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
docs: dexmate_bimanual embodiment, its null intrinsics and array rules
18e50c7 
CONTRIBUTING_DATA.md: the intrinsics rule in §6.3 and §6.4 now names the one
exception (dexmate_bimanual may store an explicit null, accepted only with
--allow-missing; an absent key always fails); §9 lists dexmate_bimanual =
8 and that ids are never renumbered; §11.1 and the checklist cover the
Dexmate checks. INTERNAL_DATA.md: ingest with --allow-missing, and the
Dexmate keypoints recipe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
dexmate: review fixes for the keymap test, spike cutoff and docs
34cc9e5 
- The moving-head keymap test now moves the hands too and checks the
  chunk's future endpoint (keypoints, palm position and palm rotation)
  against inverse(base_T_head[now]) @ base_T_palm[future], so a chunk
  that repeated the current hand configuration would fail.
- SPIKE_MAX_ABS_RAD is 6.2832, the old SharpaDex schema's max_abs, not
  2*pi: 2*pi itself no longer warns. Boundary cases added.
- dexmate_keypoints.yaml: SharpaDex always has both wrist cameras;
  turning them off also needs the wrist stems and encoders removed, since
  DDP fails on parameters with no gradient.
- CONTRIBUTING_DATA.md: the §6.4 "always a dict" note and the upload
  checklist (intrinsics, extrinsics, obs_head_pose, coordinate frame) now
  state the dexmate_bimanual exceptions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: SharpaDex manifest records and the episode ID contract
d38ccc6 
sharpadex_records holds what the converter needs from a SharpaDex manifest:
EpisodeRecord, read_manifest and ManifestError (the manifest generator stays
in the ingest tooling). It also fixes the two ID namespaces:

- the source id <task>/<season>/<episode_index> (the manifest's recording_id)
  selects and locates source episodes;
- the canonical id vendor_recording_id(task, season, index) =
  <task>__<season>__<index:06d> is the episode's vendor_recording_id, one path
  component accepted by main's vendor_recording_id_error, and parses back.

A task or season part containing "/" or "__" is rejected, so the mapping is
injective. read_manifest runs id_problems over the whole manifest (each
recording_id must match its fields, canonical ids must be valid and unique)
and turns malformed records into ManifestError. The module needs only the
stdlib. manifest_v1.jsonl (28,993 records) reads with no id problems.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: Sharpa kinematic models and FK track derivation
7c9d882 
kinematics/ holds the converter's models, shipped as package data:
- the NTH POC1.0 platform URDF and the left/right Sharpa Wave URDFs, byte for
  byte as delivered, with the hand package's LICENSE.txt and NOTICE.txt;
- sharpa_dexmate.yaml: each URDF's sha256 (checked on load), the head link,
  the hand mount, the aux joint order and all 21 MANO slot -> link mappings,
  the palm link and the tactile layout per hand;
- urdf.py: the mesh-free URDF -> pytorch-kinematics chain builder, trimmed to
  batched FK (the models have no mimic joints, so those are refused).

sharpa_tracks maps Sharpa's 65-column state/action vectors to the
dexmate_bimanual arrays and derives palm poses, MANO-21 keypoints and link
poses in robot_base. Joint names, orders and hand limits come from main's
dexmate_rules, and joint-limit excursions are labelled with the verdict main's
validator gives them (error past ERROR_MARGIN_RAD for observed hand joints,
warning past WARN_MARGIN_RAD). fk_residuals and tcp_side_check are the two
torch-side checks the converter's verification adds.

Tests check the URDF names and limits against dexmate_rules, sha256
refusal, the batched FK against pytorch-kinematics' own builder, block
independence, excursion tiers, FK residuals and the swapped-arm TCP check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: SharpaDex converter on main's writer
294ecf8 
Ports the old branch's SharpaDex profile as the only profile.

lerobot_reader reads a release's <task>/<season>/lerobot_v2.1 seasons (the
v3.0 export is not read): parquet columns as arrays, frames through PyAV with
a thread cap, and the frame-count, frame_index and global-index checks.

sharpa_to_zarr takes names, clocks and the selection from the manifest
(--tasks/--seasons/--episodes or --recording-ids) and writes through
ZarrWriter with main's required metadata: operator "sharpa" (a documented
vendor-level surrogate, one constant), lab "sharpa_dex", the task folder as
task_name, the canonical vendor_recording_id, and MissingMetadata() for
intrinsics, scene and objects. license and source_episode_id go through
metadata_override, as do the provenance attrs, the conversion report and
sharpa_array_names (joint, taxel and tile names; features stay the
writer's). Annotations go through the writer, skills through
write_annotations, before the store is renamed into place. Tactile strips
are 3-channel JPEG.

Kept from the old branch, with tests: per-hand subtask text joined with
"; " (ab7dbaa), the tactile column and mosaic layout, FFmpeg threads
capped at available_cpus() // streams (0e96505), --recording-ids and
--skip-done with ownership checks, now comparing canonical ids and checking
that source_episode_id corresponds (82c1f30), the vendor-layout and TCP
gates, and the provenance verdict (complete only without data defects).

Every episode gets <hash>.mp4 and <hash>.validation.json. The report merges
main's validate_episode(allow_missing=True), which already runs the
dexmate_bimanual rules, with the FK residuals of the stored observed and
commanded keypoints and the TCP side check; it also fails a waiver other
than scene/objects/intrinsics and ids that do not correspond. read_report
treats a missing or malformed report as a failure and recomputes ok.

Dropped with the RoCo/Origami profile: --calibration, clipping,
--utc-timestamp-column, --keep-vendor-arrays, the strict status policy,
--lerobot-version, and the string schema_version, morphology and
camera_mounts attrs. --validate and --save-mp4 are no longer options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
tests: the wheel ships the Sharpa kinematic models and they load from it
5d75559 
The wheel must contain the three URDFs, the spec, LICENSE.txt/NOTICE.txt
and the kinematics modules. A second case unpacks the wheel, runs Python
from there with only the unpacked tree on PYTHONPATH, checks that the
kinematics package is imported from it rather than the checkout, verifies
every URDF's sha256 and builds the models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: runbook for the SharpaDex converter
5d5a11a 
How to run sharpa_to_zarr on a release and a manifest, what it writes, the
two episode ids (source and canonical) and how resume and ownership use
them, the required metadata (including the operator surrogate "sharpa" and
lab "sharpa_dex"), the CC-BY-NC-4.0 licence, the array mapping, the
verification report and data_status, and the shipped kinematic models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
tests: a converted SharpaDex episode stages like any vendor episode
04f8bc4 
Runs main's stage_processed_folder.read_batch on a converted episode's
zarr.json (served by a fake S3 client): the staged row has the embodiment,
task, frame count, fps, annotations flag and start time, and raw_attrs is
plain JSON (null intrinsics/scene/objects, no NaN) with the licence and lab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: reject id parts ending in "_" so canonical ids invert…
ac7d4ec 
… exactly

vendor_recording_id("task_", "season", 1) gave "task___season__000001",
which split back as ("task", "_season", ...) and failed to parse; the same
held for a season ending in "_" (X3 finding 5). A task or season part may no
longer end in "_", which also makes the manifest-wide id_problems check
refuse such records, and the parser splits from the right.
manifest_v1.jsonl (28,993 records) still reads, and every id inverts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: fail the TCP check on non-finite poses
4c3ca91 
A NaN in observation.state.tcp_pose made position_error_m_p95 NaN, and
NaN > tolerance is false, so the check passed (X3 finding 4). tcp_side_check
now fails on non-finite TCP poses or Link_7 FK before comparing any
threshold, and never returns a non-finite metric.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: convert whole episodes and fail incomplete ones in th…
e1e6a62 
…e report

A truncated episode was marked structural_sample, but verification_report
ignored the status, so the report passed (X3 finding 3). --max-frames is
removed, as plan E drops partial-delivery options, and the reader always
reads whole episodes. The report gains a conversion check: data_status must
be complete, and the source, retained and stored frame counts must be
equal; otherwise the report fails. A swapped-TCP episode now fails it on
both counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: judge every check in read_report, not the aggregate v…
87e4e64 
…erdict

read_report returned no problems for a report without checks or
source_episode_id, and for one whose validator and TCP checks had failed,
as long as the top level said ok=True with no errors (X3 finding 2).

It now requires every field and every check (validator, fk_residual,
tcp_check, conversion), rejects NaN and Infinity, and judges each check from
its own results: no validator errors (consistent with its ok flag), all four
FK residuals finite and within tolerance, a passed TCP check whose metrics
pass the thresholds, a complete conversion with equal frame counts. ok=True
beside a failed check is a problem, as are ids that do not correspond, a
report for another episode (file name or given ids) and an unexpected
waiver. The report is written as strict JSON; a non-finite FK residual is
stored as null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: resume only finished, passing outputs; always check o…
5f19a72 
…wnership

--skip-done marked an episode done when its sidecar named it, without
opening the store or reading the report, and without --skip-done a sidecar
of another recording was never checked (X3 finding 1).

check_outputs now runs for every job. Any existing store, sidecar or report
must name the job's canonical and source ids; another recording's, or an
unreadable store, is refused even with --overwrite. A job is done only if
the store, MP4, report and sidecar all exist, the store and sidecar name it,
and the report passes read_report. Any other output of the job needs
--overwrite, and conversion removes the old sidecar and report before
writing, so a crash part-way never looks finished. The final summary counts
refused jobs (as failures, exit 1), and a converted episode's exit status
follows read_report on its fresh report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: fix the licence to cc-by-nc-4.0 and drop --license
2582a3b 
--license replaced the release licence with arbitrary text (X3 finding 6).
The converter now stores one constant, LICENSE = "cc-by-nc-4.0", the
licence the leads approved, and fails an episode whose release README
declares no licence or a different one (compared case-insensitively).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: describe the FK residual as stored-array consistency
1d1e618 
The runbook and docstrings said the FK residual confirms joint order, units
and frames. The converter derives the keypoints from the same joints under
the same column interpretation, so a permuted source would pass (X3
finding 7). The check is now described as consistency between the stored
joints, keypoints and palm poses; the TCP side check remains the only
comparison with an independent vendor signal, for the arms. A test pins the
limitation: a permuted hand block derives and checks clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: resume re-validates the store, not just its report
7f38cb6 
--skip-done trusted the passing report written at conversion time, so a
store that later lost arrays (or an empty MP4) still counted as done.
check_outputs now also runs the episode validator on the store as it is
and requires a non-empty MP4 before skipping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
tests: local Dexmate GPU train-step case
6a6a8ad 
The real SharpaDex HPT keypoints recipe takes 5 bf16 steps on one GPU over
converted episodes from EGOVERSE_TEST_DEXMATE_DIR (LocalEpisodeResolver, the
first three hashes pinned), checks the loss is finite, and rebuilds the model
from last.ckpt alone: norm stats keyed by embodiment id 8, identical weights.
Encoders start from random weights, so the test needs no network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
@jaynye @claude
sharpa_process: say where the manifest planner lives
f798aa7 
The manifest planner and the chunked ingest driver stay on the branch
jmni/sharpa-ingest-tooling as a one-off job for the v1.0 run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JEf2oCYFjeq2bxE77pt31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jaynye