Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions egomimic/hydra_configs/robot/yam_rl2_earbuds_hpt300_rollout.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,70 @@ policy:
device: cuda:0
use_ema: false
max_valid_samples: 8
# Station start values for the model's own controls (skipped when not exposed):
# sampling steps at the trained/validated 50 (ARC contracts open at 10).
inference_defaults:
inference_steps: 50
# Station-owned rollout knobs, shown on the dashboard for any model that does
# not declare a control of the same name; decoder knobs only when the loaded
# decoder has the attribute (ARC first-stream decoder; the time retimer
# declares its own speed controls). None of them changes what was trained.
station_controls:
replan_every:
label: Actions before replanning
description: Execute this prefix before predicting again (used when fastest-stream termination is 0).
type: integer
min: 1
max: 100
step: 1
default: 30
target:
kind: policy_attribute
attribute_path: replan_every
multistream_fastest_stream:
label: Fastest-stream termination
description: 1 ends the chunk when the first moving stream reaches its waypoint cap; 0 executes the fixed prefix of Actions before replanning.
type: integer
min: 0
max: 1
step: 1
default: 1
target:
kind: decoder_attribute
attribute_path: first_stream
execute_waypoint_percent:
label: Execute first % of waypoints
description: Waypoint cap while fastest-stream termination is on; must be a whole prefix of the token's waypoints.
type: integer
min: 2
max: 100
step: 1
default: 50
target:
kind: decoder_attribute
attribute_path: execute_percent
arc_speed_percent:
label: Replay speed (%)
description: Decoder replay tempo of the moving phases, not a learned speed condition.
type: integer
min: 25
max: 400
step: 5
default: 100
target:
kind: decoder_attribute
attribute_path: speed_percent
arc_hold_speed_percent:
label: Hold speed (%)
description: Replay tempo of holds (grasp dwell, placement); 100 preserves the demonstrated hold time while the moving phases change tempo.
type: integer
min: 25
max: 400
step: 5
default: 100
target:
kind: decoder_attribute
attribute_path: hold_speed_percent
adapter:
_target_: egomimic.robot.graph_policy.CartesianGraphAdapter
embodiment_id: 7
Expand Down
67 changes: 66 additions & 1 deletion egomimic/hydra_configs/robot/yam_rl2_hptflow_rollout.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,73 @@ policy:
# HPT-Flow is stochastic. Reject an invalid whole plan and resample rather
# than clamping an individual gripper; abort before robot commands if all fail.
max_valid_samples: 8
# Station start values for the model's own controls (skipped when not exposed):
# sampling steps at the trained/validated 50 (ARC contracts open at 10).
inference_defaults:
inference_steps: 50
# Station-owned rollout knobs, shown on the dashboard for any model that does
# not declare a control of the same name; decoder knobs only when the loaded
# decoder has the attribute (ARC first-stream decoder; the time retimer
# declares its own speed controls). None of them changes what was trained.
station_controls:
replan_every:
label: Actions before replanning
description: Execute this prefix before predicting again (used when fastest-stream termination is 0).
type: integer
min: 1
max: 100
step: 1
default: 30
target:
kind: policy_attribute
attribute_path: replan_every
multistream_fastest_stream:
label: Fastest-stream termination
description: 1 ends the chunk when the first moving stream reaches its waypoint cap; 0 executes the fixed prefix of Actions before replanning.
type: integer
min: 0
max: 1
step: 1
default: 1
target:
kind: decoder_attribute
attribute_path: first_stream
execute_waypoint_percent:
label: Execute first % of waypoints
description: Waypoint cap while fastest-stream termination is on; must be a whole prefix of the token's waypoints.
type: integer
min: 2
max: 100
step: 1
default: 50
target:
kind: decoder_attribute
attribute_path: execute_percent
arc_speed_percent:
label: Replay speed (%)
description: Decoder replay tempo of the moving phases, not a learned speed condition.
type: integer
min: 25
max: 400
step: 5
default: 100
target:
kind: decoder_attribute
attribute_path: speed_percent
arc_hold_speed_percent:
label: Hold speed (%)
description: Replay tempo of holds (grasp dwell, placement); 100 preserves the demonstrated hold time while the moving phases change tempo.
type: integer
min: 25
max: 400
step: 5
default: 100
target:
kind: decoder_attribute
attribute_path: hold_speed_percent
# History, shapes, decoder and typed sampler/replanning controls belong to
# the selected checkpoint's model-owned, bound inference-config.yaml.
# the selected checkpoint's model-owned, bound inference-config.yaml;
# station_controls above only fill in rollout knobs a model does not declare.
adapter:
_target_: egomimic.robot.graph_policy.CartesianGraphAdapter
embodiment_id: 7
Expand Down
64 changes: 64 additions & 0 deletions egomimic/hydra_configs/robot/yam_rl2_towels_hpt300_rollout.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,70 @@ policy:
device: cuda:0
use_ema: false
max_valid_samples: 8
# Station start values for the model's own controls (skipped when not exposed):
# sampling steps at the trained/validated 50 (ARC contracts open at 10).
inference_defaults:
inference_steps: 50
# Station-owned rollout knobs, shown on the dashboard for any model that does
# not declare a control of the same name; decoder knobs only when the loaded
# decoder has the attribute (ARC first-stream decoder; the time retimer
# declares its own speed controls). None of them changes what was trained.
station_controls:
replan_every:
label: Actions before replanning
description: Execute this prefix before predicting again (used when fastest-stream termination is 0).
type: integer
min: 1
max: 100
step: 1
default: 30
target:
kind: policy_attribute
attribute_path: replan_every
multistream_fastest_stream:
label: Fastest-stream termination
description: 1 ends the chunk when the first moving stream reaches its waypoint cap; 0 executes the fixed prefix of Actions before replanning.
type: integer
min: 0
max: 1
step: 1
default: 1
target:
kind: decoder_attribute
attribute_path: first_stream
execute_waypoint_percent:
label: Execute first % of waypoints
description: Waypoint cap while fastest-stream termination is on; must be a whole prefix of the token's waypoints.
type: integer
min: 2
max: 100
step: 1
default: 50
target:
kind: decoder_attribute
attribute_path: execute_percent
arc_speed_percent:
label: Replay speed (%)
description: Decoder replay tempo of the moving phases, not a learned speed condition.
type: integer
min: 25
max: 400
step: 5
default: 100
target:
kind: decoder_attribute
attribute_path: speed_percent
arc_hold_speed_percent:
label: Hold speed (%)
description: Replay tempo of holds (grasp dwell, placement); 100 preserves the demonstrated hold time while the moving phases change tempo.
type: integer
min: 25
max: 400
step: 5
default: 100
target:
kind: decoder_attribute
attribute_path: hold_speed_percent
adapter:
_target_: egomimic.robot.graph_policy.CartesianGraphAdapter
embodiment_id: 7
Expand Down
23 changes: 20 additions & 3 deletions egomimic/pipeline/inference_controls.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,29 @@ def public(self):
}


def configure_profile_controls(graph, training, inference_profiles, *, decoder=None):
"""Bind declared settings by stable stage ID, with atomic preflight."""
def configure_profile_controls(
graph, training, inference_profiles, *, decoder=None, station_controls=None
):
"""Bind declared settings by stable stage ID, with atomic preflight.

``station_controls`` are rollout-only settings declared by the station YAML.
Each is added only when the model declares no control of that name and, for
a decoder target, the bound decoder has that attribute; the checkpoint-bound
model declaration itself is never changed.
"""
_, profile, _ = resolve_inference_profile(training, inference_profiles)
stage_ids = OmegaConf.select(training, "model.pipeline.stage_ids", default={})
overrides = dict(profile.get("overrides", {}))
for name, spec in (station_controls or {}).items():
target = spec.get("target", {}) if isinstance(spec, Mapping) else {}
if name in overrides or (
target.get("kind") == "decoder_attribute"
and not hasattr(decoder, str(target.get("attribute_path")))
):
continue
overrides[name] = spec
bindings = []
for name, spec in profile.get("overrides", {}).items():
for name, spec in overrides.items():
validate_control(name, spec, stage_ids=stage_ids)
target = spec["target"]
kind, path = target["kind"], target["attribute_path"]
Expand Down
71 changes: 49 additions & 22 deletions egomimic/rldb/zarr/cartesian_arc.py
Original file line number Diff line number Diff line change
Expand Up @@ -441,6 +441,20 @@ def control_steps(
):
if max_steps is not None and (int(max_steps) != max_steps or max_steps < 1):
raise ValueError("max_steps must be a positive integer")
duration = self.control_duration(
token, initial_action=initial_action, clocks_in_seconds=clocks_in_seconds
)
if duration is None:
if max_steps is None:
raise ValueError(
"ARC token timing does not define a finite replan boundary"
)
return int(max_steps)
steps = max(1, int(np.ceil(duration / self.tokenizer.config.dt - 1e-9)))
return min(steps, int(max_steps)) if max_steps is not None else steps

def control_duration(self, token, *, initial_action, clocks_in_seconds=False):
"""Seconds until the first moving stream ends; None if that is undefined."""
value = self._native(token, initial_action, clocks_in_seconds)
moving_ends, static_ends = [], []
for index, name in enumerate(self._stream_names):
Expand All @@ -451,28 +465,29 @@ def control_steps(
moving_ends.append(end)
elif self.clock_mode == "duration":
static_ends.append(end)
if moving_ends:
duration = min(moving_ends)
elif static_ends:
duration = min(static_ends)
else:
if max_steps is None:
raise ValueError(
"A fully static velocity token has no duration; supply max_steps"
)
return int(max_steps)
if not np.isfinite(duration):
if max_steps is None:
raise ValueError(
"ARC token timing does not define a finite replan boundary"
)
return int(max_steps)
steps = max(1, int(np.ceil(duration / self.tokenizer.config.dt - 1e-9)))
return min(steps, int(max_steps)) if max_steps is not None else steps
ends = moving_ends or static_ends
duration = min(ends) if ends else np.inf
return float(duration) if np.isfinite(duration) else None

def _clock_indices(self, durations, action_horizon):
def stream_clocks(self, token, *, initial_action, clocks_in_seconds=False):
"""Decode-only: per arm, (arrival seconds, cumulative distance) of the
primary stream (translation if present), each of length M + 1."""
value = self._native(token, initial_action, clocks_in_seconds)
anchor = _initial_actions(initial_action)
names = self._stream_names
name = "translation" if "translation" in names else names[0]
index = names.index(name)
out = []
for arm, off in enumerate((0, 7)):
distance = self._distances(value, off, name, anchor)
durations = self._intervals(distance, value[:, 14 + index * 2 + arm])
out.append(
(np.r_[0.0, np.cumsum(durations)], np.r_[0.0, np.cumsum(distance)])
)
return out

def _clock_indices(self, durations, query):
elapsed = np.r_[0.0, np.cumsum(durations)]
query = self.tokenizer.config.dt * np.arange(1, action_horizon + 1)
upper = np.clip(
np.searchsorted(elapsed, query, side="right"), 1, len(elapsed) - 1
)
Expand All @@ -488,15 +503,27 @@ def _clock_indices(self, durations, action_horizon):
return lower, upper, np.clip(alpha, 0.0, 1.0), elapsed

def detokenize(
self, arc_actions, action_horizon, *, initial_action, clocks_in_seconds=False
self,
arc_actions,
action_horizon,
*,
initial_action,
clocks_in_seconds=False,
times=None,
):
"""``times`` (decode-only): token-clock seconds to read the H rows at,
replacing the uniform next-frame grid (dt, 2 dt, ..., H dt)."""
value = self._native(arc_actions, initial_action, clocks_in_seconds)
h = int(action_horizon)
if h < 1:
raise ValueError("action_horizon must be positive")
anchor = _initial_actions(initial_action)
geometry = np.concatenate((anchor[None], value[:, :14]), axis=0)
query = np.arange(1, h + 1) * self.tokenizer.config.dt
if times is not None:
query = np.asarray(times, dtype=np.float64)
if query.shape != (h,) or not np.isfinite(query).all():
raise ValueError(f"times must be {h} finite clock readings")
arms = []
for arm, off in enumerate((0, 7)):
indices, clocks = {}, {}
Expand All @@ -505,7 +532,7 @@ def detokenize(
self._distances(value, off, name, anchor),
value[:, 14 + index * 2 + arm],
)
lower, upper, alpha, elapsed = self._clock_indices(durations, h)
lower, upper, alpha, elapsed = self._clock_indices(durations, query)
indices[name] = (lower, upper, alpha)
clocks[name] = elapsed
primary = indices.get("translation", indices.get("rotation"))
Expand Down
5 changes: 5 additions & 0 deletions egomimic/robot/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,11 @@ for Eva/Yam setup, Quest app build requirements, data format and operator comman
Two, four or six timing channels follow all 14 pose channels; they are not interleaved.
Decoder settings belong to explicit model YAML `decoder_attribute` controls;
never append controls based on a model class or inferred token width. The
station YAML's `policy.station_controls` adds rollout-only knobs (replan,
fastest stream, waypoint %, replay/hold speed) where the model declares none
of that name and the loaded decoder has the target attribute, and
`policy.inference_defaults` sets start values of exposed controls; neither
edits the checkpoint-bound declaration. The
decoders use the live `egomimic.rldb.zarr` tokenizers only; no checkpoint-era
codec copies are carried. Anything the codec needs for deployment
(``detokenize(times=)``, `stream_clocks`, `stream_ends`) is decode-only; never
Expand Down
Loading
Loading