Hi authors, thank you for releasing WorldMemArena.
Could you clarify the automated dataset-construction pipeline, particularly for the synthetic Lifelong Evolution split?
The paper states that sessions are generated from hidden world states, gold memory points are extracted and refined, and QA checkpoints are constructed from those memory points. However, we could not find the following information in the paper, dataset card, or repository:
- Which model(s) were used to generate the Lifelong sessions?
- Which model(s) and prompts were used to extract/merge/revise gold memory points?
- Which model(s) and prompts were used to generate questions and gold answers?
- Were the automatic validators model-based or rule-based?
- What did the 2–3 human annotators review or modify? Are annotation guidelines, edit rates, or agreement statistics available?
- Are there plans to release the construction scripts, prompts, model versions, and generation parameters?
This provenance seems important for reproducing the benchmark and assessing possible generator–annotator–judge bias, especially because the Lifelong dialogues, reference memory points, and QA pairs may be produced within the same automated pipeline.
Thanks!
Hi authors, thank you for releasing WorldMemArena.
Could you clarify the automated dataset-construction pipeline, particularly for the synthetic Lifelong Evolution split?
The paper states that sessions are generated from hidden world states, gold memory points are extracted and refined, and QA checkpoints are constructed from those memory points. However, we could not find the following information in the paper, dataset card, or repository:
This provenance seems important for reproducing the benchmark and assessing possible generator–annotator–judge bias, especially because the Lifelong dialogues, reference memory points, and QA pairs may be produced within the same automated pipeline.
Thanks!