How to eval on Emu edit benchmark

[Eureka-Maggie]

Hi,

Noticed that there are known issues with the Emu edit benchmark: some image-caption pairs seem incorrect (e.g., 'a train station in city') or identical source and target captions. So I was wondering how to calculate clip_dir metric. How did you process the benchmark dataset?

Looking forward to your reply.

[wusize]

Hi! I also have the same question. Can the authors provide more details of the evaluation on emu edit test?

[AFeng-x]

Hi, sorry for not responding to you promptly. We did not process the captions; and we used the original benchmark without any modification.