Is it reasonable for the teacher observations to include the reference trajectory?

[Ba-Lance]

We are distilling the pnp_table teacher into a vision student that only consumes head-camera RGB plus proprioception, and ran into a question about the observation design.

Besides the privileged object state, the teacher's policy observation group also contains the reference trajectory: command_multi_future and similar terms (which are also the encoder's input).

The reference trajectory is an externally supplied intent and timing signal. The student does not have it at deployment, and it cannot be inferred from RGB either.

Two questions:

If the goal is an RGB + proprio student, should the teacher have been trained with proprioception plus object/table state only, dropping the reference-trajectory terms?
If the teacher does need the reference trajectory, what is the recommended way to close this gap? Is there an existing RGB-only student config we could look at?

[Divadi]

I believe it is okay for the teacher to observe the reference motion, since the teacher is fundamentally a motion tracking policy alongside HOI objectives. The intuition I went with was to treat the teacher like teleop - the teacher has in its mind the table/object state, alongside the intended motion, and the student learns to plan reasonable motion given limited observations.

"If the goal is an RGB + proprio student, should the teacher have been trained with proprioception plus object/table state only, dropping the reference-trajectory terms?" - I think this is possible, and it ties into recent work like HIL, but we did not explore it in this work.
"If the teacher does need the reference trajectory, what is the recommended way to close this gap? Is there an existing RGB-only student config we could look at?" - the student should not need the reference trajectory as input during inference, just the RGB and proprio.

[Ba-Lance]

Thanks for the reply! I have a couple of follow-up questions:

First, on treating the teacher as teleop and going the VLA route:
In this view, a natural way is to save the teacher rollouts as a LeRobot-format sim dataset, and then fine-tune a VLA like GR00T. Just want to double-check: if all episodes are the same type of task, is a generic prompt like "pick up the object on the table" good enough? Or is it better to write it as "pick up the {object}"? (The latter feels like it would naturally resolve any ambiguity about "which object to grab".)

Second, regarding "reasonable motion":
I take this to mean that the action labels are multimodal relative to the student's observation (i.e., the same RGB + proprio can map to different phases or styles of the reference trajectory). If so, doing MSE regression on the teacher's pre-quant latents seems problematic—MSE assumes a unimodal Gaussian, so with multimodality, it will just average things out, leading to blurry and unsafe actions.

So I would like to ask what the right treatment is here:

Do we keep privileged distillation but swap the step-by-step MSE for a multimodal loss? Like running cross-entropy on discrete tokens after FSQ, or using a flow matching / diffusion action head?
Or does this actually go beyond imitation learning, meaning we need RL? Personally, I lean towards thinking RL is better as a fine-tuning phase after BC/DAgger, rather than replacing distillation altogether—since the teacher gives us dense supervision basically for free. Does that make sense?