WorldFailBench Dataset

This repository documents the open-source dataset for the paper:

WorldFailBench: Do World Models Understand Policy Failures?

WorldFailBench evaluates whether world models can serve as faithful policy evaluators — not only predicting whether a robot policy succeeds or fails, but also how it fails (failure reason), when it fails (failure stage), and under what conditions it transitions from success to failure (failure boundary). This dataset provides the simulator ground truth used to score world-model behavior: real closed-loop rollouts executed in the RoboTwin and LIBERO physics simulators, together with success/failure labels, task-stage annotations, and controlled success/failure boundary cases.

Dataset Download

Download the dataset from Hugging Face: ACERobotics/WorldFailBench.

Direct download: dataset.tar.gz

After downloading dataset.tar.gz, extract it with:

tar -xzf dataset.tar.gz

The archive extracts into a dataset/ directory with the structure below.

Dataset Overview

Subset	Simulator	Contents	Scale
`RoboTwin/`	RoboTwin	First-frame image + metadata per evaluation episode	5 policy configurations × 20 tasks × 50 episodes = 5,000 episodes
`LIBERO/`	LIBERO (libero_10)	First-frame image (+ raw array) + scene metadata per evaluation episode	3 policy variants × 10 tasks × 50 episodes = 1,500 episodes
`boundary/`	RoboTwin + LIBERO	Paired success/failure boundary cases built by step-wise perturbation of the initial scene, with full rollout videos	120 pairs, 2,400 rollout episodes

dataset/
├── RoboTwin/                     # RoboTwin evaluation episodes
│   ├── pi05/                     # π0.5 policy
│   ├── pi05-data-20/             # π0.5 fine-tuned with 20% of the training data
│   ├── pi05-data-50/             # π0.5 fine-tuned with 50% of the training data
│   ├── fastwam/                  # FastWAM policy
│   ├── lingbot-vla-v2/           # LingBot-VLA v2 policy
│   │   └── <task_name>/          # 20 dual-arm tasks per policy
│   │       ├── episode_000/
│   │       │   ├── first_frame.png
│   │       │   └── metadata.json
│   │       ├── episode_001/
│   │       └── ...               # 50 episodes per task
│   └── ...
├── LIBERO/                       # LIBERO evaluation episodes
│   ├── pi05_data20/              # π0.5 fine-tuned with 20% of the training data
│   ├── pi05_data50/              # π0.5 fine-tuned with 50% of the training data
│   ├── pi05_data100/             # π0.5 fine-tuned with 100% of the training data
│   │   └── task_XX/              # 10 LIBERO-10 tasks per policy variant
│   │       ├── episode_000/
│   │       │   ├── first_rgb.png
│   │       │   ├── first_rgb.npy
│   │       │   └── scene.json
│   │       ├── episode_001/
│   │       └── ...               # 50 episodes per task
│   └── ...
└── boundary/                     # Success/failure boundary cases
    ├── LIBERO/
    │   ├── manifest.json         # LIBERO pair list + selection criteria
    │   ├── orientation/          # pair_001 – pair_020   (20 pairs)
    │   ├── position/             # pair_021 – pair_040   (20 pairs)
    │   └── size/                 # pair_041 – pair_060   (20 pairs)
    └── RoboTwin/
        ├── position/             # pair_001 – pair_020   (20 pairs)
        ├── orientation/          # pair_021 – pair_040   (20 pairs)
        └── size/                 # pair_041 – pair_060   (20 pairs)

1. Evaluation Episodes (`RoboTwin/` and `LIBERO/`)

Each top-level folder groups episodes by policy (configuration) and then by task. Every episode records the first frame of the rollout (the initial scene observation) and a JSON file with ground-truth metadata. These episodes support the Failure Reason and Failure Stage evaluation tasks described in the paper.

1.1 RoboTwin episodes

RoboTwin/<policy>/<task_name>/episode_XXX/

first_frame.png — RGB image of the initial scene (first frame of the rollout).
metadata.json — ground-truth metadata:

{
  "task": "open_laptop",
  "seed": 200000,
  "success": false,
  "instruction": "Pull the lid of the black laptop with smooth matte surface upward.",
  "split": "seen",
  "stages": ["Reach", "Grasp", "Lift"]
}

Field	Description
`task`	RoboTwin task name (e.g. `open_laptop`, `place_cans_plasticbox`).
`seed`	Random seed used to generate the initial scene.
`success`	Ground-truth outcome of the rollout in the simulator (`true`/`false`).
`instruction`	Natural-language task instruction given to the policy.
`split`	Generalization split of the scene (`seen`).
`stages`	Ordered task-specific stage decomposition used for failure-stage annotation (full lists in Section 3).

Policies: pi05, pi05-data-20, pi05-data-50, fastwam, lingbot-vla-v2.
20 tasks per policy, 50 episodes per task → 5,000 episodes in total (1,774 success / 3,226 failure).

1.2 LIBERO episodes

LIBERO/<policy_variant>/task_XX/episode_XXX/

first_rgb.png — RGB image of the initial scene (first frame of the rollout).
first_rgb.npy — the same first frame as a raw NumPy array.
scene.json — ground-truth metadata:

{
  "task": "LIVING_ROOM_SCENE2_put_both_the_alphabet_soup_and_the_tomato_sauce_in_the_basket",
  "init_state_id": 0,
  "language": "put both the alphabet soup and the tomato sauce in the basket",
  "suite": "libero_10",
  "success": false,
  "task_id": 0,
  "stages": [
    "Grasp the alphabet soup.",
    "Move the alphabet soup to the basket.",
    "Release the alphabet soup and retreat the arm.",
    "Grasp the tomato sauce.",
    "Move the tomato sauce to the basket.",
    "Release the tomato sauce and retreat the arm."
  ]
}

Field	Description
`task`	Full LIBERO task name (scene + instruction).
`task_id`	Integer task index within the suite (`task_00`–`task_09`).
`suite`	LIBERO suite (`libero_10`, a.k.a. LIBERO-Long).
`init_state_id`	Index of the initial state used for the rollout.
`language`	Natural-language task instruction.
`success`	Ground-truth outcome of the rollout (`true`/`false`).
`stages`	Ordered task-specific stage decomposition used for failure-stage annotation (full lists in Section 3).

Policy variants: pi05_data20, pi05_data50, pi05_data100 (π0.5 fine-tuned with 20% / 50% / 100% of the training data).
10 LIBERO-10 tasks per variant, 50 episodes per task → 1,500 episodes in total (1,056 success / 444 failure).

2. Failure Type Taxonomy

The Failure Reason task annotates every confirmed failure with one primary failure type, adapting the planning/execution distinction of REFLECT and Guardian to trajectory annotation. Labels describe observable behavior in the context of the task instruction, without assuming internal policy causes. There are three labels:

Label	Definition	Examples
Task Planning Failure	Task-inappropriate choices of targets, subgoals, or their ordering — e.g. omitting a required step, or attempting a subgoal before its prerequisites are satisfied.	Closing a cabinet before inserting the required object, then attempting insertion through the closed door (violates a task dependency even if the closing motion itself succeeds).
Action Execution Failure	Failure to carry out an intended subgoal through motion or physical interaction.	Missing a grasp (gripper closes beside the mug handle and lifts without the mug); dropping a held object during transport; inaccurate placement.
Indeterminate Failure (`Unjudgable`)	A confirmed failure whose primary type cannot be identified reliably, e.g. because the decisive interaction is occluded. Recorded as `Unjudgable`; these episodes remain in the all-episode denominator.	—

Annotation rules (see Appendix B of the paper):

For each confirmed failure, annotators assign one primary type and one decisive stage; when several errors occur, the type follows the earliest clearly supported primary cause, and the stage identifies the earliest decisive event preventing completion.
Stage names are copied verbatim from the task-specific stage lists below; if a stage cannot be identified, annotators record it as unknown.
Successful episodes contribute zero failure counts.

3. Task Stage Decomposition

The stages field of every episode lists the ordered semantic stages of its task. These task-specific lists define the vocabulary of failure stages: stage-level failure annotations always reference one of these stages (or unknown). The same decomposition is shared by all policies for a given task.

3.1 RoboTwin tasks (20)

Stages of dual-arm tasks are prefixed with the acting arm (Arm 1: / Arm 2:); / separates stages executed by the two arms in parallel.

Task	Ordered stages
`beat_block_hammer`	Reach → Grasp → Transfer → Strike
`blocks_ranking_rgb`	Reach → Grasp red → Transfer right → Place → Grasp green → Transfer center → Grasp blue → Transfer left
`blocks_ranking_size`	Reach → Grasp small → Transfer right → Place → Grasp medium → Transfer center → Grasp large → Transfer left
`handover_block`	Arm 1: Reach → Arm 1: Grasp → Arm 1: Transfer → Arm 2: Reach → Arm 2: Grasp → Arm 1: Release → Arm 2: Transfer → Arm 2: Place
`hanging_mug`	Arm 1: Reach → Arm 1: Grasp → Arm 1: Place → Arm 2: Reach → Arm 2: Grasp → Arm 2: Transfer → Arm 2: Place
`move_pillbottle_pad`	Reach → Grasp → Transfer → Place
`open_laptop`	Reach → Grasp → Lift
`pick_diverse_bottles`	Arm 1: Reach / Arm 2: Reach → Arm 1: Grasp / Arm 2: Grasp → Arm 1: Transfer / Arm 2: Transfer
`place_a2b_left`	Reach → Grasp A → Place A
`place_bread_skillet`	Arm 1: Reach / Arm 2: Reach → Arm 1: Grasp / Arm 2: Grasp → Rotate the skillet → Transfer the bread → Place the bread
`place_cans_plasticbox`	Arm 1: Reach / Arm 2: Reach → Arm 1: Grasp / Arm 2: Grasp → Arm 1: Transfer → Arm 1: Place → Arm 1: Retract → Arm 2: Transfer → Arm 2: Place → Arm 2: Retract
`place_dual_shoes`	Arm 1: Reach / Arm 2: Reach → Arm 1: Grasp / Arm 2: Grasp → Arm 1: Transfer → Arm 1: Place → Arm 1: Retract → Arm 2: Transfer → Arm 2: Place
`place_empty_cup`	Reach → Grasp → Transfer → Place
`place_object_scale`	Reach → Grasp → Transfer → Place
`place_phone_stand`	Reach → Grasp → Transfer → Place
`put_bottles_dustbin`	Bottle 1: Reach → Bottle 1: Grasp → Bottle 1: Transfer → Bottle 1: Place → Bottle 2: Reach → Bottle 2: Grasp → Bottle 2: Transfer → Bottle 2: Place → Bottle 3: Reach → Bottle 3: Grasp → Bottle 3: Transfer → Bottle 3: Place
`put_object_cabinet`	Arm 1: Reach → Arm 1: Grasp → Arm 2: Reach → Arm 2: Grasp → Arm 2: Pull open → Arm 1: Transfer → Arm 1: Place
`scan_object`	Arms 1 and 2: Reach → Arms 1 and 2: Grasp → Arm 1: Transfer → Arm 2: Transfer
`stack_blocks_three`	Block 1: Reach → Block 1: Grasp → Block 1: Place → Block 1: Retract → Block 2: Reach → Block 2: Grasp → Block 2: Place → Block 2: Retract → Block 3: Reach → Block 3: Grasp → Block 3: Place → Block 3: Retract
`stack_bowls_three`	Bowl 1: Reach → Bowl 1: Grasp → Bowl 1: Place → Bowl 1: Retract → Bowl 2: Reach → Bowl 2: Grasp → Bowl 2: Place → Bowl 2: Retract → Bowl 3: Reach → Bowl 3: Grasp → Bowl 3: Place → Bowl 3: Retract

3.2 LIBERO-10 tasks (10)

Task	Ordered stages
task_00	Grasp the alphabet soup → Move the alphabet soup to the basket → Release the alphabet soup and retreat the arm → Grasp the tomato sauce → Move the tomato sauce to the basket → Release the tomato sauce and retreat the arm
task_01	Grasp the cream cheese box → Move the cream cheese box to the basket → Release the cream cheese box and retreat the arm → Grasp the butter → Move the butter to the basket → Release the butter and retreat the arm
task_02	Turn on the stove → Grasp the moka pot → Move the moka pot to the stove → Place the moka pot on the stove, release it, and retreat the arm
task_03	Open the bottom drawer → Grasp the black bowl → Move the black bowl into the bottom drawer → Release the bowl and retreat the arm → Close the bottom drawer and retreat the arm
task_04	Grasp the white mug → Move the white mug to the left plate → Release the white mug and retreat the arm → Grasp the yellow-and-white mug → Move the yellow-and-white mug to the right plate → Release the mug and retreat the arm
task_05	Grasp the book → Move the book to the back compartment of the caddy → Release the book and retreat the arm
task_06	Grasp the white mug → Move the white mug to the plate → Release the mug and retreat the arm → Grasp the chocolate pudding → Move the chocolate pudding to the right of the plate → Release the pudding and retreat the arm
task_07	Grasp the alphabet soup → Move the alphabet soup to the basket → Release the soup and retreat the arm → Grasp the cream cheese box → Move the cream cheese box to the basket → Release the box and retreat the arm
task_08	Grasp the first moka pot → Move the first moka pot to the stove → Release the first moka pot and retreat the arm → Grasp the second moka pot → Move the second moka pot to the stove → Release the second moka pot and retreat the arm
task_09	Open the microwave door → Grasp the yellow-and-white mug → Move the mug into the microwave → Release the mug and retreat the arm → Close the microwave door and retreat the arm

4. Boundary Cases (`boundary/`)

The boundary/ subset supports the Failure Boundary evaluation task. Each pair is constructed by starting from a base initial scene and perturbing a single scene dimension — object position, orientation, or size — in small steps until the policy's outcome flips. A pair therefore contains two adjacent variants that straddle the success→failure transition:

success/ — rollouts from the last scene variant that (mostly) succeeds;
failure/ — rollouts from the first scene variant where performance collapses.

Unlike the evaluation episodes above, every boundary episode also includes the full rollout video, enabling frame-level comparison against world-model generations.

boundary/<Simulator>/<dimension>/pair_XXX/
├── metadata.json                 # pair-level boundary description
├── success/
│   ├── episode_000/              # LIBERO sides: episode_000 - episode_009
│   │   ├── rollout.mp4           # full rollout video
│   │   └── scene.json            # episode-level record
│   └── ...                       # 10 episodes per side
│                                 # (RoboTwin sides are numbered episode_001 - episode_010)
├── failure/
│   └── ...
└── success_scene.json            # (RoboTwin only) full scene snapshot of the success variant
└── failure_scene.json            # (RoboTwin only) full scene snapshot of the failure variant

Counts: RoboTwin 60 pairs (position / orientation / size × 20) with 10 rollouts per side; LIBERO 60 pairs (orientation / position / size × 20) with 10 rollouts per side → 2,400 boundary rollout episodes in total. The pair counts match the paper exactly: 60 frozen boundary pairs per platform, 20 per perturbation factor (Appendix C). The LIBERO pair list and selection criteria are recorded in boundary/LIBERO/manifest.json.

4.1 Episode-level record (`scene.json`)

RoboTwin example:

{
  "policy": "FastWAM",
  "task": "place_cans_plasticbox",
  "language": "Lift the hand-sized metallic can and the white patterned brown metal can, and stick them into the rectangular storage box with grooves",
  "variant_label": "plasticbox_size+0pct",
  "perturb_dimension": "Size",
  "success": false
}

LIBERO example:

{
  "policy": "pi05_head_finetuned",
  "task": "KITCHEN_SCENE3_turn_on_the_stove_and_put_the_moka_pot_on_it",
  "language": "turn on the stove and put the moka pot on it",
  "variant_idx": 2,
  "variant_label": "moka_yaw+20",
  "perturb_dimension": "orientation",
  "perturb_delta": 20.0,
  "success": true,
  "gripper_action_conversion": "unit_to_native"
}

Field	Description
`policy`	Policy that was rolled out.
`task` / `language`	Task name and natural-language instruction.
`variant_label`	Human-readable label of the perturbed scene variant (e.g. `moka_yaw+20`, `plasticbox_size+20pct`).
`variant_idx`	(LIBERO) Index of the variant along the perturbation sweep.
`perturb_dimension`	Perturbed dimension: `position`, `orientation`, or `size`.
`perturb_delta`	(LIBERO) Magnitude of the perturbation relative to the base scene.
`success`	Observed outcome of this rollout.
`gripper_action_conversion`	(LIBERO) Gripper action space conversion used by the policy.

4.2 Pair-level record (`metadata.json`)

LIBERO example:

{
  "task_id": 2,
  "base_episode": 0,
  "perturb_joint": "moka_pot_1_joint0",
  "perturb_axis": 0,
  "dimension": "position",
  "success_variant": 0,
  "failure_variant": 1,
  "success_delta": 0.0,
  "failure_delta": 0.02,
  "success_label": "moka_x+0cm",
  "failure_label": "moka_x+2cm",
  "success_count": 10,
  "failure_count": 7,
  "success_rate": 1.0,
  "failure_rate": 0.7,
  "asset_id": "libero/pi05_libero_data20"
}

Field	Description
`task_id` / `base_episode`	Task index and base episode from which the sweep started.
`perturb_joint` / `perturb_axis`	Perturbed object joint and axis.
`dimension`	Perturbed dimension (`position` / `orientation` / `size`).
`success_variant`, `failure_variant`	Variant indices forming the boundary pair.
`success_delta`, `failure_delta`	Perturbation magnitudes of the two variants.
`success_label`, `failure_label`	Human-readable labels of the two variants.
`success_count`, `failure_count`	Number of successful rollouts observed on each side.
`success_rate`, `failure_rate`	Observed success rate on each side of the boundary.
`asset_id`	Identifier of the policy variant used for the sweep (e.g. `libero/pi05_libero_data20`).

RoboTwin pair records use a different schema (they do not contain the LIBERO-specific fields above such as base_episode, perturb_joint, success_variant or asset_id). Example (RoboTwin/size/pair_041):

{
  "policy": "FastWAM",
  "policy_id": "fastwam",
  "task_id": 26,
  "task": "place_cans_plasticbox",
  "dimension": "Size",
  "category": "size",
  "object_axis": "plasticbox_size",
  "success_label": "plasticbox_size+0pct",
  "failure_label": "plasticbox_size+20pct",
  "success_rate_high": 0.7,
  "success_rate_low": 0.2,
  "drop": 0.5,
  "endpoint_gap": {
    "previous_value": 1.0,
    "failed_value": 1.2,
    "signed_change": 0.2,
    "absolute_change": 0.2,
    "unit": "ratio",
    "failed_over_previous_size_ratio": 1.2,
    "relative_linear_size_change_percent": 20.0,
    "relative_linear_size_change_absolute_percent": 20.0,
    "scale_factor_reference": "original object linear dimensions"
  }
}

Field	Description
`policy`, `policy_id`	Policy that was rolled out (`FastWAM` / `Pi0.5` / `LingBot-VLA-v2`) and its identifier.
`task_id`, `task`	RoboTwin task index and name.
`dimension`	Perturbed dimension, capitalized: `Position`, `Orientation`, or `Size`.
`category`	Lowercase category of the perturbation: `position`, `rotation` (for orientation), or `size`.
`object_axis`	Perturbed scene axis (e.g. `plasticbox_size`).
`success_label`, `failure_label`	Labels of the two boundary variants.
`success_rate_high`, `success_rate_low`	Observed success rates of the success-side and failure-side variants (10 rollouts each).
`drop`	Success-rate drop across the boundary (`success_rate_high` − `success_rate_low`).
`endpoint_gap`	Physical change between the two variants. Its keys are dimension-specific: position pairs include `absolute_change_cm`, orientation pairs include `absolute_change_degrees`, and size pairs include the size-ratio fields shown above.

4.3 Scene snapshots (RoboTwin only)

success_scene.json / failure_scene.json store the complete simulator scene state (object poses, geometry, etc., under snapshot.objects) together with the perturbed actor, feature, value, unit, and instruction, so that both boundary scenes can be exactly re-instantiated in the RoboTwin simulator.

5. Intended Use

The dataset provides the simulator ground truth for the three WorldFailBench evaluation tasks:

Failure Reason — given the first frame and instruction of a failed episode (RoboTwin/, LIBERO/), evaluate whether a world model's rollout reproduces the observable error the real policy makes, using the failure type taxonomy in Section 2.
Failure Stage — using the task-specific stage decompositions (Section 3) stored in each episode's stages field, evaluate whether failures in world-model rollouts concentrate at the same execution stages as in the simulator.
Failure Boundary — using the paired variants in boundary/, evaluate whether a world model reproduces the success→failure transition under the same controlled scene change.

6. Notes

All episodes are closed-loop rollouts of real VLA policies executed in the physics simulators, not in world models; they serve as reference ground truth.
success labels are produced by each simulator's task success checker.
Failure-type and failure-stage labels follow the annotation protocol of the paper: stage names are copied from the task-specific lists in Section 3, and unidentifiable stages are recorded as unknown; unclassifiable failures are recorded as Unjudgable.
Boundary pairs are labeled by the relative outcome change across the pair; the absolute success rate on the failure side may still be non-zero (see failure_rate in the pair metadata).

7. Citation

If you find this dataset useful, please cite:

@article{worldfailbench,
  title  = {WorldFailBench: Do World Models Understand Policy Failures?},
  author = {Zhang, Zheng and Tang, Xiaoyang and Cheng, Shuaiqi and Huang, Jiakai and Wu, Zhongbo and Zhang, Ziyi and Si, Yuan and Liu, Zeyu and Gao, Weibo and Liu, Shunyu and You, Shan and Wang, Fei and Wang, Xiaogang and Tao, Dacheng},
  year   = {2026}
}

8. License

This dataset is released under the Apache License 2.0. The full license text is available in LICENSE.

Name		Name	Last commit message	Last commit date
LICENSE		LICENSE
README.md		README.md

Navigation Menu

Repository navigation

About

Repository files navigation

WorldFailBench Dataset

Dataset Download

Dataset Overview

1. Evaluation Episodes (`RoboTwin/` and `LIBERO/`)

1.1 RoboTwin episodes

1.2 LIBERO episodes

2. Failure Type Taxonomy

3. Task Stage Decomposition

3.1 RoboTwin tasks (20)

3.2 LIBERO-10 tasks (10)

4. Boundary Cases (`boundary/`)

4.1 Episode-level record (`scene.json`)

4.2 Pair-level record (`metadata.json`)

4.3 Scene snapshots (RoboTwin only)

5. Intended Use

6. Notes

7. Citation

8. License

About

Releases

Packages

Contributors

Latest commit

History

Folders and files

Repository files navigation

WorldFailBench Dataset

Dataset Download

Dataset Overview

1. Evaluation Episodes (RoboTwin/ and LIBERO/)

1.1 RoboTwin episodes

1.2 LIBERO episodes

2. Failure Type Taxonomy

3. Task Stage Decomposition

3.1 RoboTwin tasks (20)

3.2 LIBERO-10 tasks (10)

4. Boundary Cases (boundary/)

4.1 Episode-level record (scene.json)

4.2 Pair-level record (metadata.json)

4.3 Scene snapshots (RoboTwin only)

5. Intended Use

6. Notes

7. Citation

8. License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

1. Evaluation Episodes (`RoboTwin/` and `LIBERO/`)

4. Boundary Cases (`boundary/`)

4.1 Episode-level record (`scene.json`)

4.2 Pair-level record (`metadata.json`)