Bug in camera action processing

[georgysavva]

Hi, thank you for open-sourcing such an important research dataset! We've been using your dataset to build our Minecraft world model.

We took your dataloading code as is, and I believe I found a bug in the camera action post-processing logic. It implicitly converts any camera values to integers, so numbers like 5.625 become 5. Here I added a test that shows it: link. It outputs:

Mismatched elements: 2 / 2 (100%)
Max absolute difference: 0.625
Max relative difference: 0.11111111
 x: array([5, 5])
 y: array([5.625, 5.625], dtype=float32)

Meaning the function returns 5 whereas it should have been 5.625 == 37.5 * 360/2400. The bug stems from the way the camera numpy array is initialized here and here. It gets created as an integer array so that float numbers that later get assigned get converted to integers. It should be np.array([0.0, 0.0], dtype=np.float32) instead. Here is the fix: georgysavva@4ac7eb0. With it, the above test passes.

The implication of this bug is that camera actions get very inaccurate at lower values. For example, the relative error for 2.9 and 2.0 is 45%. Here is a histogram of the camera action distribution of the VPT dataset (X axis is the value in degrees):

As you can see, most of the action values are in lower ranges, < 3.0, so the issue is well represented in the dataset.

[Miffyli]

Heya! Nice catch! I think this has gone unnoticed for so long since the camera movement is thought to be defined in "pixels" for mouse movement, although I can not remember if this is exactly the case for Minecraft (i.e., does not treat mouse movement in full pixels or would floating numbers work as well).

It would be misleading to store the numbers in floats if they are still treated as integers down the line (or worse; do not work). Can you check what the impact is on the game by e.g., running sequence of 1.4 actions vs. 1 vs. 1.7 (to see if there is rounding)?

For world models I understand they do not really care about this, but since this codebase is originally for playing Minecraft, I'd avoid breaking/confusing the logic there.

[georgysavva]

By "running sequence" you mean executing various camera degree values in MineRL to see if the game itself is sensitive to 1.7 or 1.0 degree values?

I am coming from a world model project, and it's important for world models as well:

The error gets to almost 50% for the 2.0 and 2.9 for the camera value pair. This discrepancy means that the action the world model sees and the actual observation change won't be corresponding. The action for any of the values between 2.0 and 2.9 will be the same, but actual observation change will differ.

[Miffyli]

By "running sequence" you mean executing various camera degree values in MineRL to see if the game itself is sensitive to 1.7 or 1.0 degree values?

Yes indeed!

I am coming from a world model project, and it's important for world models as well:

Yup I agree. However since this project was done before the world models popularity, and is oriented around the Minecraft game playing (the Minecraft itself), I am inclined to keep the logic compatible with the game itself. A world model project should fork the code and adjust as needed for their purposes, with the understand that executing action "1.7" in the world model will have different results to running the actual game, where "1.7" may be simply mapped to "1.0".

[jxiong21029]

From what I can tell, Malmo uses integer mouse movements (see here) when low level inputs are enabled, which seems standard, so camera movement are quantized to 0.15 degree increments, at least for the (pitch, yaw) minerec format.

However, the integer minerec (pitch, yaw) is mapped to degrees when converting from the minerec to minerl format (see here) by multiplying by CAMERA_SCALER=360/2400, so I believe quantizing the minerl camera action to integers does actually result in a precision loss.

[Miffyli]

Yeap definitely there is precision loss, no doubt about it! My original argument was about keeping "compatibility" with the MineRL environment, which I thought accepts integers as camera control input (incorrectly), but looking over the code now, it seems to indeed accept floats. My bad, been too long since worked on this stuff 😅 .

Happy to approve changes/tests that fix these, e.g., stuff in @georgysavva 's fork.

Very cool work on Solaris btw @georgysavva ! Cool to see Minecraft still being active research platform 😎