A "time alignment" issue that I think the author overlooked（一个我认为作者疏忽的”时间对齐“问题）

[Love-syntacticSugar]

In the forward method of SOLOFusion-main\mmdet3d\models\necks\view_transformer_solofusion.py, there is the following code:

SOLOFusion/mmdet3d/models/necks/view_transformer_solofusion.py

Lines 311 to 314 in 683edce

	with torch.set_grad_enabled(curr_stereo_feats.requires_grad):
	curr_curr_stereo_feats_valid_list = [ # List of size B, inner is _ x C. ignore 118 dimension for now; it's only needed when indexing into cost volume.
	tmp[tmp_where[0], :, tmp_where[2], tmp_where[3]] for tmp, tmp_where in zip(curr_stereo_feats, curr_valid_mask_where_list)]
	curr_curr_stereo_feats_valid_padded = pad_sequence(curr_curr_stereo_feats_valid_list, batch_first=True) # B x max_length x C

I believe the author overlooked the issue of "temporal alignment" here. The qualified point coordinates obtained from the variable curr_valid_mask_where_list are currently in the "previous frame" image coordinate system. How can the author use the point coordinates from the "previous frame" image coordinate system to extract corresponding feature vectors from the "current frame" backbone feature map?

I think the correct approach should be:

1. Based on the result of curr_valid_mask with shape (B, N, 7, stereo_H, stereo_W,), obtain the coordinates of each qualified point in the "previous frame" image coordinate system, resulting in (B, N, 7, stereo_H, stereo_W, 4) with (x, y, d, 1).
   (Note: Since some points at certain depths may be unqualified, the number of qualified points corresponding to each candidate point may vary. Therefore, in practice, a mask value is also needed to record which points are qualified.)

2. Then, use matrix operations to perform the following transformations:
   "Previous frame" image coordinate system → "Previous frame" camera coordinate system → "Previous frame" global coordinate system → "Current frame" global coordinate system → "Current frame" camera coordinate system → "Current frame" image coordinate system.
   The result at this point would be (B, N, 7, stereo_H, stereo_W, 4) with (x, y, d, 1), representing the coordinates of the qualified points in the "current frame" image coordinate system.

3. Only then can the corresponding feature vectors be extracted from the "current frame" backbone feature map.

Finally, this question has troubled me for a long time. I wonder if I have described it clearly. I also hope that the author can give an answer.Thank you very much! !

The following is the Chinese translation:
我认为这里作者忘记考虑了"时间对齐"的问题，变量curr_valid_mask_where_list所得到的合格点坐标此时处于”上一帧“的img坐标系下，作者怎么能用”上一帧“的img坐标系下的点坐标去”当前帧“的backbone特征图中取出对应的特征向量呢？
我认为正确做法应该是：

1. 根据curr_valid_mask的结果(B, N, 7, stereo_H, stereo_W, )，获取每一个合格点在”上一帧“img坐标系下的坐标(B, N, 7, stereo_H, stereo_W, 4)，(x,y,d,1)
   (注意：因为有些深度上的点可能是不合格的点，所以每一个候选点对应的合格点的数量还不一样。所以这里实际实现起来，也需要一个mask值，记录哪一些点是合格点）
2. 然后，利用矩阵运算实现：
   ”上一帧“img坐标系-->”上一帧“camera坐标系-->”上一帧“global坐标系-->”当前帧“global坐标系-->
   ”当前帧“camera坐标系-->”当前帧“img坐标系
   此时得到的结果：(B, N, 7, stereo_H, stereo_W, 4)，(x,y,d,1)，才是当前帧的img坐标系下的合格点坐标
3. 然后才能去”当前帧“的backbone特征图中取出对应的特征向量

最后，这个问题困扰我许久，不知道我是否描述清楚了？也希望作者可以给出解答，万分感谢！！