Adding an Entity

In the implementation (Section 4) of the paper, we discuss two classes of mutation, Changing Object Color (Section 4.2) and Adding Entities (Section 4.3). This page discusses additional algorithmic details for the implementation of the Adding an Entity mutation.

Algorithm

The main algorithm is implemented in image_mutator.py in the function add_instance(semantic_label, bg_id, add_id, rotation), available here.

Parameters

The function parameters are:

• semantic_label - a string indicating the semantic label of the entity to add. This must be a class recognized by the underlying dataset. The default value is car as used for the 'Add Car' mutation, but person is also explored for the 'Add Pedestrian' mutation.
• bg_id - the unique identifier of the background image to add the entity to (the destination image). This identifier corresponds to the identifier used in the underlying dataset. This parameter defaults to None, in which case the algorithm will select a destination image at random. Specifying this parameter is mainly used to reproduce a previously performed mutation.
• add_id - the unique identifier of the entity to add. This identifier corresponds to the identifier used in the underlying dataset. This parameter defaults to None, in which case the algorithm will select an entity semi-randomly while ensuring conformity. Specifying this parameter is mainly used to reproduce a previously performed mutation.
• rotation - Unused in the study, this parameter was briefly explored as an option to expand the space of available mutations by allowing added entities to be rotated from their original position. Defaults to None, if specified, gives the amount to rotate the entity before adding, in radians.

Conformity Checking

1. Randomly select a destination image (bg_id) if one is not already provided (here)
2. Randomly select an entity to add (here) and then check, in order:
   1. Is the entity a single continuous entity? If not, select again. (here: len(contours) != 1)
   2. Is the entity sufficiently large to be interesting? If not, select again. This checks the dimensions of the bounding box. For car this is at least 100px on the longer side, for a person this is at least 50px on the longer side. (here)
   3. [Only for cars] Based on the information in the dataset, can we determine that there are any other entities that are both in front of the selected entity and overlap the selected entity? If so, the entity is occluded and not complete, select again. (here)
   4. Will adding the entity result in the entity being on the road in the destination image? If not, select again. This is checked by determining if the lower left of the entity would be on the road. (here)
   5. Check to see if the lighting is similar between the entity and the location where it is to be added in the destination image. If not, select again. Function call here. Logic checks if the median value from HSV color space of the selected regions are within 5.
   6. Will the entity be in the correct perspective if it is added? If not, select again. Function call here. Logic checks to see if the vanishing point of the scene is in the same quadrant.