The issues in measure-melting-point

[TonyStark042]

Hi, thanks for your contribution on ScienceWorld, and I found serval issues in this.

1. The gold Path generated by env.getGoldActionSequence() doesn't make sense. For example:

• action: use thermometer in inventory on chocolate, obs: the thermometer measures a temperature of 34 degrees celsius。
• action: focus on green box, obs: You focus on the green box.

The generated action and the corresponding obs did not observe any change in the state of substance and directly selected the answer.

2. The problem in measuring the melting point of mercury
   First, all the observation got by the gold actions still don't have a change in the state of substance, but it still choose the correct answer.
   Moreover, the freezer seems very difficult to freeze the mercury to its true melting point of -35 celsius, i tried aound 50 round and it still didn't work.

[PeterAJansen]

Hi @TonyStark042 , thanks for your issue report.

For these issues, can you provide code to replicate them (including the environment initialization, the task, the environment seed, and so forth?).

For the second issue, not being able to get to -35C in the freezer -- were you using the kitchen freezer (which gets to -18C, as a normal kitchen freezer), or the ultra-cold lab freezer in the workshop?

[TonyStark042]

Hi, thanks for your reply. Below is the code for reproducing the issue, i am using scienceworld==1.1.3. You can find the issue almost in all the variations.
By the way, some other tasks have such similar issues in some extent. For example, the task 0 (boil) gold path has sequence like this:

• action: examine substance in metal pot, obs: a substance called water
• action: use thermometer in inventory on substance in metal pot, obs: the thermometer measures a temperature of 96 degrees celsius
• action: examine steam, obs: a substance called steam
  Obviously, there is no evidence indicating that steam was produced, yet the action chooses to directly examine the steam, which is unreasonable. You can verify this using the same code.

env = ScienceWorldEnv("", envStepLimit=1000)
data_type = "train"

env.load("measure-melting-point-known-substance", generateGoldPath=True)
if data_type == "train":
    vari_ids = env.getVariationsTrain()
elif data_type == "dev":
    vari_ids = env.getVariationsDev()
elif data_type == "test":
    vari_ids = env.getVariationsTest()
path = f"dataset/scienceworld/expert_path/task{task_id}/"
os.makedirs(path, exist_ok=True)
for vari in vari_ids:
    env.load("measure-melting-point-known-substance", vari, generateGoldPath=True)
    task_description = env.taskdescription()
    gold_data = {
        "task_description":task_description,
        "action":[],
        "obs":[],
        "next_obs":[],
        "reward":[],
        "score":[],
        "done":[],
    }

    obs, _ = env.reset()
    gold_data["action"] = env.getGoldActionSequence()
    for action in gold_data["action"]:
        obs_, reward, isCompleted, infos = env.step(action)
        gold_data["obs"].append(obs)
        gold_data["reward"].append(reward/100)
        gold_data["score"].append(infos["score"]/100)
        gold_data["done"].append(isCompleted)
        gold_data["next_obs"].append(obs_)
        obs = obs_
    
    with open(path+f'variation{vari}.json', 'w') as json_file:
        json.dump(gold_data, json_file, indent=4)
        print(task_id, vari,"   done")

For the second issue, all the gold action sequences in measuring the melting point of mercury task select the freezer in kitechen, this mislead me and the subsquent model fine-tuning.

[PeterAJansen]

Hi @TonyStark042 ,

I think I see the issue. At each step, the agent essentially has three observations available to it -- the normal obs, but also the look (environmental observation) and inv (inventory observation). Certainly if just looking at obs then the observation space could seem to be a little incomplete, unless you frequently also input the look around and inventory commands.

I think another issue (from our end) is that the bots that generate the gold action sequences are slightly inconsistent in their use of adding these performative simulated look around commands. So there are (often, but not 100% of the time) explicit look around and examine command that call attention to the items that should be attended to, but clearly this isn't always the case, as in the chocolate example (or the steam example). For example, if we modify your code to simulate what the agent would see in plain text:

import os
import json
from scienceworld import ScienceWorldEnv

env = ScienceWorldEnv("", envStepLimit=1000)
data_type = "train"

env.load("measure-melting-point-known-substance", generateGoldPath=True)
if data_type == "train":
    vari_ids = env.getVariationsTrain()
elif data_type == "dev":
    vari_ids = env.getVariationsDev()
elif data_type == "test":
    vari_ids = env.getVariationsTest()
#path = f"dataset/scienceworld/expert_path/task{task_id}/"
#os.makedirs(path, exist_ok=True)
for vari in vari_ids:
    env.load("measure-melting-point-known-substance", vari, generateGoldPath=True)
    task_description = env.taskdescription()
    gold_data = {
        "task_description": task_description,
        "subtask": [],
        "action": [],
        "obs": [],
        "look": [],
        "inventory": [],
        "next_obs": [],
        "reward": [],
        "score": [],
        "done": [],
    }

    obs, _ = env.reset()
    gold_data["action"] = env.getGoldActionSequence()
    for action in gold_data["action"]:
        obs_, reward, isCompleted, infos = env.step(action)
        gold_data["obs"].append(obs)
        gold_data["look"].append(infos["look"])
        gold_data["inventory"].append(infos["inv"])
        gold_data["reward"].append(reward / 100)
        gold_data["score"].append(infos["score"] / 100)
        gold_data["done"].append(isCompleted)
        gold_data["next_obs"].append(obs_)
        obs = obs_

    with open(f'variation{vari}.json', 'w') as json_file:
        json.dump(gold_data, json_file, indent=4)
        #print(task_id, vari,"   done")
        print("Variation " + str(vari) + " completed.")

    # Generate a string version
    out_str = ""
    out_str += "Task Description: " + task_description + "\n\n"

    # Number of steps = number of actions
    num_steps = len(gold_data["action"])
    for idx in range(num_steps):
        out_str += "\n"
        out_str += "--------------------------------------------------------------\n"        
        out_str += "Step " + str(idx) + ":\n"        
        out_str += "--------------------------------------------------------------\n"        
        out_str += "\n"
        
        out_str += "Obs: " + str(gold_data["obs"][idx]) + "\n"
        out_str += "\n"
        
        if ((idx-1) >= 0):
            out_str += "Look: " + str(gold_data["look"][idx-1]) + "\n"
            out_str += "Inventory: " + str(gold_data["inventory"][idx-1]) + "\n"
        
        out_str += ">>>>> Action: " + str(gold_data["action"][idx]) + "\n"
        out_str += "\n"
        
        out_str += "\n"
        out_str += "Reward: " + str(gold_data["reward"][idx]) + "\n"
        out_str += "Score: " + str(gold_data["score"][idx]) + "\n"
        out_str += "Done: " + str(gold_data["done"][idx]) + "\n"
        #out_str += "Next Obs: " + str(gold_data["next_obs"][idx]) + "\n"
        
    with open(f'variation{vari}.txt', 'w') as f:
        f.write(out_str)

Then we do see the gold sequence providing an explicit measure temperature (to see the object's current temperature) and examine X (to see if the object's description has changed, signifying a change in state) loop. But, at the end of this loop (variation 0), the obs still says chocolate, where as the look notes that it's turned into liquid chocolate, so the bot exits out of the loop and makes the decision. In these cases, the gold bot should likely wait an additional iteration so that it includes a clear performative examine that returns the expected state change, so that it's more transparent in the gold sequence, and not only in the look. (Highlighted as *** in Step 20):

--------------------------------------------------------------
Step 18:
--------------------------------------------------------------

Obs: the thermometer measures a temperature of 25 degrees celsius

Look: This room is called the kitchen. In it, you see: 
	the agent
	a substance called air
	a blue box (containing nothing)
	a chair. On the chair is: nothing.
	a counter. On the counter is: a bowl (containing a red apple, a banana, an orange, a potato), a drawer.
	a cupboard. The cupboard door is open. In the cupboard is: a ceramic cup (containing nothing), a drawer, a tin cup (containing nothing).
	a freezer. The freezer door is closed. 
	a fridge. The fridge door is open. In the fridge is: a wood cup (containing orange juice).
	a glass jar (containing a substance called sodium chloride)
	a lighter
	a orange box (containing nothing)
	a oven, which is turned off. The oven door is closed. 
	a painting
	a sink, which is turned off. In the sink is: nothing.
	a substance called soap
	a stopwatch, which is deactivated. 
	a stove, which is turned on. On the stove is: a metal pot (containing chocolate).
	a table. On the table is: a glass cup (containing nothing).
You also see:
	A door to the bathroom (that is closed)
	A door to the hallway (that is open)
	A door to the outside (that is closed)

Inventory: In your inventory, you see:
	an orange
	a thermometer, currently reading a temperature of 10 degrees celsius

>>>>> Action: examine chocolate


Reward: 0.0
Score: 0.78
Done: False

--------------------------------------------------------------
Step 19:
--------------------------------------------------------------

Obs: chocolate

Look: This room is called the kitchen. In it, you see: 
	the agent
	a substance called air
	a blue box (containing nothing)
	a chair. On the chair is: nothing.
	a counter. On the counter is: a bowl (containing a red apple, a banana, an orange, a potato), a drawer.
	a cupboard. The cupboard door is open. In the cupboard is: a ceramic cup (containing nothing), a drawer, a tin cup (containing nothing).
	a freezer. The freezer door is closed. 
	a fridge. The fridge door is open. In the fridge is: a wood cup (containing orange juice).
	a glass jar (containing a substance called sodium chloride)
	a lighter
	a orange box (containing nothing)
	a oven, which is turned off. The oven door is closed. 
	a painting
	a sink, which is turned off. In the sink is: nothing.
	a substance called soap
	a stopwatch, which is deactivated. 
	a stove, which is turned on. On the stove is: a metal pot (containing liquid chocolate).
	a table. On the table is: a glass cup (containing nothing).
You also see:
	A door to the bathroom (that is closed)
	A door to the hallway (that is open)
	A door to the outside (that is closed)

Inventory: In your inventory, you see:
	an orange
	a thermometer, currently reading a temperature of 10 degrees celsius

>>>>> Action: use thermometer in inventory on chocolate


Reward: 0.01
Score: 0.79
Done: False

--------------------------------------------------------------
Step 20:
--------------------------------------------------------------

Obs: the thermometer measures a temperature of 35 degrees celsius

Look: This room is called the kitchen. In it, you see: 
	the agent
	a substance called air
	a blue box (containing nothing)
	a chair. On the chair is: nothing.
	a counter. On the counter is: a bowl (containing a red apple, a banana, an orange, a potato), a drawer.
	a cupboard. The cupboard door is open. In the cupboard is: a ceramic cup (containing nothing), a drawer, a tin cup (containing nothing).
	a freezer. The freezer door is closed. 
	a fridge. The fridge door is open. In the fridge is: a wood cup (containing orange juice).
	a glass jar (containing a substance called sodium chloride)
	a lighter
	a orange box (containing nothing)
	a oven, which is turned off. The oven door is closed. 
	a painting
	a sink, which is turned off. In the sink is: nothing.
	a substance called soap
	a stopwatch, which is deactivated. 
***	a stove, which is turned on. On the stove is: a metal pot (containing *****liquid***** chocolate).
	a table. On the table is: a glass cup (containing nothing).
You also see:
	A door to the bathroom (that is closed)
	A door to the hallway (that is open)
	A door to the outside (that is closed)

Inventory: In your inventory, you see:
	an orange
	a thermometer, currently reading a temperature of 10 degrees celsius

>>>>> Action: focus on blue box


Reward: 0.21
Score: 1.0
Done: True

--------------------------------------------------------------
Step 21:
--------------------------------------------------------------

Obs: You focus on the blue box.

Look: This room is called the kitchen. In it, you see: 
	the agent
	a substance called air
	a blue box (containing nothing)
	a chair. On the chair is: nothing.
	a counter. On the counter is: a bowl (containing a red apple, a banana, an orange, a potato), a drawer.
	a cupboard. The cupboard door is open. In the cupboard is: a ceramic cup (containing nothing), a drawer, a tin cup (containing nothing).
	a freezer. The freezer door is closed. 
	a fridge. The fridge door is open. In the fridge is: a wood cup (containing orange juice).
	a glass jar (containing a substance called sodium chloride)
	a lighter
	a orange box (containing nothing)
	a oven, which is turned off. The oven door is closed. 
	a painting
	a sink, which is turned off. In the sink is: nothing.
	a substance called soap
	a stopwatch, which is deactivated. 
	a stove, which is turned on. On the stove is: a metal pot (containing liquid chocolate).
	a table. On the table is: a glass cup (containing nothing).
You also see:
	A door to the bathroom (that is closed)
	A door to the hallway (that is open)
	A door to the outside (that is closed)

Inventory: In your inventory, you see:
	an orange
	a thermometer, currently reading a temperature of 10 degrees celsius

>>>>> Action: wait1


Reward: 0.0
Score: 1.0
Done: True

In the mean time, I would recommend including the look and inv keys as part of the observation to your agent, which should still provide the agent the information it needs to determine which action to take -- just sometimes not as obvious as it could be.

[TonyStark042]

Thanks a lot! This solves my problem.
How about the second issue? The gold action sequence lead the agent to the freezer in kitchen instead of the ultra-cold lab freezer in the workshop in measuring the melting point of mercury task.