Summary
examples/assembly_station.py generates a great asymmetric-visibility scene: an
overview camera frames the operator (COCO-17) and a worktop camera frames the
part_* items, with complementary per-camera visible labels. A downstream
consumer (multicam-occlusion's fusion mode) already uses this end-to-end for
order verification: route entities by visibility, reconstruct assembled
contents from the item camera's placements, reconcile against the emitted
order.json BOM → fulfilled / missing / wrong / extra. That half runs on real
generated output today. 🎉
The other half of that mode — causal action↔assembly-change association
("that operator reach caused that part's placement") — cannot yet run on
generated data. The current operator does a single continuous sinusoidal wrist
reach; it is not authored to dip a hand synced to each part placement. So
there are no per-placement action events to correlate in time, and the
consumer keeps that metric on a hand-authored controlled scene.
Ask (help wanted)
Let the assembly-station operator (or the pose/entity layer generally) emit
placement-synced action events: a discrete hand-dip (a strict local minimum
of a tracked keypoint's height at the station) timestamped just before each
item's placement frame, plus optionally a distractor reach that places
nothing and an item that moves outside a causal lag window. Concretely, a preset
where:
- the operator's tracked hand keypoint dips at frames
t_i - δ for each placed
item i (so a reach-and-return local minimum is recoverable off the manifest);
- one extra dip assembles nothing (distractor action);
- one item moves late, outside the lag window (distractor change);
- ground-truth
(actor, item, action_time, change_time) pairs are emitted as a
sidecar (e.g. interactions.json) so a consumer can score precision/recall
without re-deriving the truth.
This would let the causal association metric run on real multicam-sim output
too, closing the second half of the fusion integration. Overlaps naturally with
the skeletal-motion DSL (#29: reach/wave gaits) — a "reach-to-place" gait keyed
to placement frames is exactly this.
Not asking for
No schema change to manifest.json is required for the visibility half — that
already works. This is about authoring operator motion + action ground truth
timed to placements.
Summary
examples/assembly_station.pygenerates a great asymmetric-visibility scene: anoverview camera frames the
operator(COCO-17) and a worktop camera frames thepart_*items, with complementary per-cameravisiblelabels. A downstreamconsumer (multicam-occlusion's fusion mode) already uses this end-to-end for
order verification: route entities by visibility, reconstruct assembled
contents from the item camera's placements, reconcile against the emitted
order.jsonBOM → fulfilled / missing / wrong / extra. That half runs on realgenerated output today. 🎉
The other half of that mode — causal action↔assembly-change association
("that operator reach caused that part's placement") — cannot yet run on
generated data. The current operator does a single continuous sinusoidal wrist
reach; it is not authored to dip a hand synced to each part placement. So
there are no per-placement action events to correlate in time, and the
consumer keeps that metric on a hand-authored controlled scene.
Ask (help wanted)
Let the assembly-station operator (or the pose/entity layer generally) emit
placement-synced action events: a discrete hand-dip (a strict local minimum
of a tracked keypoint's height at the station) timestamped just before each
item's placement frame, plus optionally a distractor reach that places
nothing and an item that moves outside a causal lag window. Concretely, a preset
where:
t_i - δfor each placeditem
i(so a reach-and-return local minimum is recoverable off the manifest);(actor, item, action_time, change_time)pairs are emitted as asidecar (e.g.
interactions.json) so a consumer can score precision/recallwithout re-deriving the truth.
This would let the causal association metric run on real multicam-sim output
too, closing the second half of the fusion integration. Overlaps naturally with
the skeletal-motion DSL (#29: reach/wave gaits) — a "reach-to-place" gait keyed
to placement frames is exactly this.
Not asking for
No schema change to
manifest.jsonis required for the visibility half — thatalready works. This is about authoring operator motion + action ground truth
timed to placements.