Evaluation on V-COCO

[fredzzhang]

We use the evaluation utilities provided by Gupta et al. Note that although the utilities were developed in Python2, we recommend installing it with Python3. You might need change a few lines of code, but it significantly speeds up the evaluation.

Instructions

1. Navigate to the root directory of the repository v-coco.
2. Save the following script in a file named utils.py

from collections import defaultdict

class CacheTemplate(defaultdict):
    """A template for VCOCO cached results """
    def __init__(self, **kwargs):
        super().__init__()
        for k, v in kwargs.items():
            self[k] = v
    def __missing__(self, k):
        seg = k.split('_')
        # Assign zero score to missing actions
        if seg[-1] == 'agent':
            return 0.
        # Assign zero score and a tiny box to missing <action,role> pairs
        else:
            return [0., 0., .1, .1, 0.]

3. Save the following script in a file with a name of your choice. Change the variable det_file to match the path of your cached file and run the script.

from vsrl_eval import VCOCOeval
import utils

if __name__ == "__main__":
    vsrl_annot_file = "data/vcoco/vcoco_test.json"
    coco_file = "data/instances_vcoco_all_2014.json"
    split_file = "data/splits/vcoco_test.ids"

    # Change this line to match the path of your cached file
    det_file = "./cache.pkl"

    print(f"Loading cached results from {det_file}.")
    vcocoeval = VCOCOeval(vsrl_annot_file, coco_file, split_file)
    vcocoeval._do_eval(det_file, ovr_thresh=0.5)

4. The computed average precision includes a class (point-instr) that does not correspond to valid human-object interactions, which has been excluded from the final performance as a convention. To get the true performance, a simple way is to multiply the computed value by 25 and then divide by 24.