Skip to content

Assembly-station: emit placement-synced operator action events (for causal action↔change fusion) #34

Description

@bamdadd

Summary

examples/assembly_station.py generates a great asymmetric-visibility scene: an
overview camera frames the operator (COCO-17) and a worktop camera frames the
part_* items, with complementary per-camera visible labels. A downstream
consumer (multicam-occlusion's fusion mode) already uses this end-to-end for
order verification: route entities by visibility, reconstruct assembled
contents from the item camera's placements, reconcile against the emitted
order.json BOM → fulfilled / missing / wrong / extra. That half runs on real
generated output today. 🎉

The other half of that mode — causal action↔assembly-change association
("that operator reach caused that part's placement") — cannot yet run on
generated data. The current operator does a single continuous sinusoidal wrist
reach; it is not authored to dip a hand synced to each part placement. So
there are no per-placement action events to correlate in time, and the
consumer keeps that metric on a hand-authored controlled scene.

Ask (help wanted)

Let the assembly-station operator (or the pose/entity layer generally) emit
placement-synced action events: a discrete hand-dip (a strict local minimum
of a tracked keypoint's height at the station) timestamped just before each
item's placement frame, plus optionally a distractor reach that places
nothing and an item that moves outside a causal lag window. Concretely, a preset
where:

  • the operator's tracked hand keypoint dips at frames t_i - δ for each placed
    item i (so a reach-and-return local minimum is recoverable off the manifest);
  • one extra dip assembles nothing (distractor action);
  • one item moves late, outside the lag window (distractor change);
  • ground-truth (actor, item, action_time, change_time) pairs are emitted as a
    sidecar (e.g. interactions.json) so a consumer can score precision/recall
    without re-deriving the truth.

This would let the causal association metric run on real multicam-sim output
too, closing the second half of the fusion integration. Overlaps naturally with
the skeletal-motion DSL (#29: reach/wave gaits) — a "reach-to-place" gait keyed
to placement frames is exactly this.

Not asking for

No schema change to manifest.json is required for the visibility half — that
already works. This is about authoring operator motion + action ground truth
timed to placements.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions