Hi authors,
Thanks for sharing this fantastic work and open-sourcing the code! I am very interested in your evaluation metrics, specifically PresentQuiz and IP Memory.
I noticed the evaluation scripts are available in the repository (e.g., under src/evaluation/PresentQuiz and IPMemory), and the main benchmark dataset is hosted on Hugging Face (ZaynZhu/Paper2Video).
Could you please provide a bit more detail on the following?
-
PresentQuiz Questions: Are the exact multiple-choice questions (QA sets) used for the 101 papers available in the Hugging Face dataset? If so, could you point me to the specific JSON/file format?
-
IP Memory Data: For the IP memory evaluation, how are the 5-second video clips and the corresponding queries structured in the dataset?
If there are any specific system prompts or intermediate generated data (like the generated question pairs from construct.py) that you could share as a quick example, it would be extremely helpful for reproducing the evaluation!
Thanks again for the great contribution to the community!
Hi authors,
Thanks for sharing this fantastic work and open-sourcing the code! I am very interested in your evaluation metrics, specifically PresentQuiz and IP Memory.
I noticed the evaluation scripts are available in the repository (e.g., under
src/evaluation/PresentQuizandIPMemory), and the main benchmark dataset is hosted on Hugging Face (ZaynZhu/Paper2Video).Could you please provide a bit more detail on the following?
PresentQuiz Questions: Are the exact multiple-choice questions (QA sets) used for the 101 papers available in the Hugging Face dataset? If so, could you point me to the specific JSON/file format?
IP Memory Data: For the IP memory evaluation, how are the 5-second video clips and the corresponding queries structured in the dataset?
If there are any specific system prompts or intermediate generated data (like the generated question pairs from construct.py) that you could share as a quick example, it would be extremely helpful for reproducing the evaluation!
Thanks again for the great contribution to the community!