Can’t reproduce SoccerNetGS test results

[Lehsuby]

Can’t reproduce SoccerNetGS test results

Description

I’m unable to reproduce the results reported in the paper using the SoccerNetGS 2024 test set and the Jaccard@5 metric from the sn-calibration repository.

Steps to Reproduce

1. Build the Docker image

   docker build . -t broadtrack

2. Run the container (mounting the dataset and scripts)

   docker run --rm -it --gpus=all \
     -v SoccerNetGS:/root/SoccerNetGS \
     -v broadtrack/scripts:/root/scripts \
     broadtrack /bin/bash

3. Install Python and dependencies

   apt-get update && apt-get install -y python3 python3-pip
   pip install -r /root/scripts/requirements.txt

4. For each video folder:
   a. Initial run with camera parameters (X=0, Y=55, Z=-12)

   broadtrack --f /root/SoccerNetGS/test/<video_folder>/img1 \
     --X 0 --Y 55 --Z -12 \
     --o /root/SoccerNetGS/test/<video_folder>/output/json_video.json

   b. Compute tripod parameters

   python /root/scripts/compute_tripod.py \
     -i /root/SoccerNetGS/test/<video_folder>/output/json_video.json \
     -o /root/SoccerNetGS/test/<video_folder>/output/json_tripod.json

   c. Re-run with computed tripod parameters

   broadtrack --f /root/SoccerNetGS/test/<video_folder>/img1 \
     --t /root/SoccerNetGS/test/<video_folder>/output/json_tripod.json \
     --o /root/SoccerNetGS/test/<video_folder>/output/json_video_with_tripod.json

5. Convert json to camera params from sn-calibration

   class Camera:
       def __init__(self, calib_json_object):
           """
           Loads camera parameters from dictionary.
           :param calib_json_object: the dictionary containing camera parameters.
           """
           self.image_width = calib_json_object["sensorResolutionWidthPixels"]
           self.image_height = calib_json_object["sensorResolutionHeightPixels"]
           self.principal_point = (self.image_width/2., self.image_height/2. )
           hfov = calib_json_object["horizontalFieldOfViewDegrees"]*np.pi/180.
           focal = 2 * self.principal_point[0] / (2 * np.tan(hfov / 2.))
           self.xfocal_length = focal
           self.yfocal_length = focal
   
           self.calibration = np.array([
               [self.xfocal_length, 0, self.principal_point[0]],
               [0, self.yfocal_length, self.principal_point[1]],
               [0, 0, 1]
           ], dtype='float')
   
           self.pan = calib_json_object['panDegrees']
           self.tilt = calib_json_object['tiltDegrees']
           self.roll = calib_json_object['rollDegrees']
           pan = calib_json_object['panDegrees'] * np.pi / 180.
           tilt = calib_json_object['tiltDegrees'] * np.pi / 180.
           roll = calib_json_object['rollDegrees'] * np.pi / 180.
   
           self.rotation = np.array([
               [-np.sin(pan) * np.sin(roll) * np.cos(tilt) + np.cos(pan) * np.cos(roll),
                np.sin(pan) * np.cos(roll) + np.sin(roll) * np.cos(pan) * np.cos(tilt), np.sin(roll) * np.sin(tilt)],
               [-np.sin(pan) * np.cos(roll) * np.cos(tilt) - np.sin(roll) * np.cos(pan),
                -np.sin(pan) * np.sin(roll) + np.cos(pan) * np.cos(roll) * np.cos(tilt), np.sin(tilt) * np.cos(roll)],
               [np.sin(pan) * np.sin(tilt), -np.sin(tilt) * np.cos(pan), np.cos(tilt)]
           ], dtype='float')
   
           self.rotation = np.transpose(self.pan_tilt_roll_to_orientation(pan, tilt, roll))
   
           self.position = np.array([
               calib_json_object['positionXMeters'],
               calib_json_object['positionYMeters'],
               calib_json_object['positionZMeters']
                                     ], dtype='float')
   
           normalized_radial_distortion_k1 = calib_json_object['normalizedRadialDistortionCoefficients'][0]
           normalized_focal = self.image_height / self.yfocal_length
           self.radial_distortion_k1 = normalized_radial_distortion_k1 / (normalized_focal ** 2)
   
       def to_json_parameters(self):
           return {
               "pan_degrees": self.pan,
               "tilt_degrees": self.tilt,
               "roll_degrees": self.roll,
               "x_focal_length": self.calibration[0, 0],
               "y_focal_length": self.calibration[1, 1],
               "principal_point": [self.principal_point[0], self.principal_point[1]],
               "position_meters": [self.position[0], self.position[1], self.position[2]],
               "rotation_matrix": [[self.rotation[0, 0], self.rotation[0, 1], self.rotation[0, 2]],
                                 [self.rotation[1, 0], self.rotation[1, 1], self.rotation[1, 2]],
                                 [self.rotation[2, 0], self.rotation[2, 1], self.rotation[2, 2]]],
               "radial_distortion": [self.radial_distortion_k1, 0., 0., 0., 0., 0.],
               "tangential_distortion": [0., 0.],
               "thin_prism_distortion": [0., 0., 0., 0.]
           }
   
       def get_look_at_vector(self):
           return self.rotation[2]
   
       @staticmethod
       def pan_tilt_roll_to_orientation(pan, tilt, roll):
           """
           Conversion from euler angles to orientation matrix.
           :param pan:
           :param tilt:
           :param roll:
           :return: orientation matrix
           """
           Rpan = np.array([
               [np.cos(pan), -np.sin(pan), 0],
               [np.sin(pan), np.cos(pan), 0],
               [0, 0, 1]])
           Rroll = np.array([
               [np.cos(roll), -np.sin(roll), 0],
               [np.sin(roll), np.cos(roll), 0],
               [0, 0, 1]])
           Rtilt = np.array([
               [1, 0, 0],
               [0, np.cos(tilt), -np.sin(tilt)],
               [0, np.sin(tilt), np.cos(tilt)]])
           rotMat = np.dot(Rpan, np.dot(Rtilt, Rroll))
           return rotMat

6. Evaluate with sn-calibration

Observed Behavior

• I commented out the block around line 111 in compute_tripod.py (to bypass the assertion) because it fails:

  Traceback (most recent call last):
    File "/root/scripts/compute_tripod.py", line 259, in <module>
      analyse_sequence(args.input, args.output)
    File "/root/scripts/compute_tripod.py", line 177, in analyse_sequence
      assert len(intersect_optical_axes_match)
  AssertionError
  

• Final metrics differ significantly from those in the paper:

  {
    "completeness": 1.0,
    "meanRecall": 0.44478709,
    "meanPrecision": 0.10040818,
    "meanAccuracies": 0.09859953, # Jaccard @ 5
    "finalScore": 0.09859953075647354
  }

Expected Behavior

The reported Jaccard@5 metrics on the SoccerNetGS 2024 test set should match the values presented in the paper.

Environment

• BroadTrack: main branch (as of February 17, 2025)
• sn-calibration: main branch (as of June 18, 2024)
• Python: 3.10.12
• GPU: NVIDIA 4090
• Driver Version: 570.133.20
• CUDA Version: 12.8

Additional Context

Any guidance on required preprocessing, parameter tuning, or code adjustments to reproduce the published results would be greatly appreciated.

[fmagera]

Hi @Lehsuby,

Thank you for your interest in our work!

I see two possible reasons:

1. In the paper, we filter out the players from the optical flow correspondences. To do so, we detect them in frames using RTMDet which is included in mmpose.
2. If the assertion fails, there is a problem. Can you show the camera parameters provided in input? Moreover, the code provided here is made to try our method on any sequence, if you're targetting specifically sn-gamestate dataset, you might want to leverage the fact that different sequences correspond to the same camera.

[Lehsuby]

Hello @fmagera,

Thank you for your clarifications and advice. I have two follow-up questions:

1. Where to find the player-filtering code?

Could you please point me to the file or module where the step of removing players from the optical-flow correspondences (using RTMDet in mmpose) is implemented? I’d like to understand exactly how you filter out the player bounding boxes before further processing.

2. Per-video parameter workflow

For all videos, I used the same three-step pipeline:
a. Initial run with camera parameters (X=0, Y=55, Z=-12)

broadtrack --f /root/SoccerNetGS/test/<video_folder>/img1 \
  --X 0 --Y 55 --Z -12 \
  --o /root/SoccerNetGS/test/<video_folder>/output/json_video.json

b. Compute tripod parameters

python /root/scripts/compute_tripod.py \
  -i /root/SoccerNetGS/test/<video_folder>/output/json_video.json \
  -o /root/SoccerNetGS/test/<video_folder>/output/json_tripod.json

c. Re-run with computed tripod parameters

broadtrack --f /root/SoccerNetGS/test/<video_folder>/img1 \
  --t /root/SoccerNetGS/test/<video_folder>/output/json_tripod.json \
  --o /root/SoccerNetGS/test/<video_folder>/output/json_video_with_tripod.json

In other words, I executed this cycle separately for each video, so the resulting camera parameters likely differ between clips. Am I correct that, for sn-gamestate, it’s better to fix the camera parameters across the entire dataset (as you suggested) rather than re-estimating them for each clip?

I’m attaching my results for two example videos (at each of the three steps) for your review.

I’d appreciate any further guidance!

json_video_SNGS-117.json
json_video_SNGS-116.json
json_tripod_SNGS-116.json
json_tripod_SNGS-117.json


json_video_with_tripod_SNGS-116.json
json_video_with_tripod_SNGS-117.json

[fmagera]

Hi @Lehsuby ,

Here's the bounding boxes extracted for the sequence SNGS-116:
human-bboxes.zip
The bounding boxes are loaded here: https://github.com/evs-broadcast/BroadTrack/blob/main/main.cpp#L53 and optical flow correspondences are filtered in this loop: https://github.com/evs-broadcast/BroadTrack/blob/main/main.cpp#L391

No, it is not the same tripod for the whole dataset. Here's the mapping between sequences and games.
match_info.json
And here's the corresponding tripod values I obtained:
tripod_info_oleg.json.

Anyway, even without player filtering and tripod values, you should have much better results. From what I see in the .json files, it does not look that bad. I would expect something like 40% JaC@5, not 10 :) Can you share your evaluation code?

[Lehsuby]

@fmagera,

Thank you very much for the provided files and information. I will try running the pipeline with these new inputs.

I’m also sharing my evaluation Jupyter notebook for your reference:

evaluation.ipynb.zip

And can you write plan, how reproduce your results?
Is it correct plan?

• Initial run with camera parameters (X=0, Y=55, Z=-12), player-filtration (--b option, I can use SoccernetGS bboxes gt) on combined videos (concat folders by match_id)
• Compute tripod parameters for every combined videos
• Re-run with computed tripod parameters with tripod params (--t option), player-filtration (--b option, I can use SoccernetGS bboxes gt) on every video folder (not concat folders by match_id)

Best regards!

[fmagera]

In fact the last step is enough: run with the tripod positions I gave you, use the player bounding boxes from the annotations, and yes run for each sequence independently as they may not be contiguous in time.

The eval code looks good, but there's a mismatch between the sn-calibration and sn-gamestate pitch labels to account for :

corrected_pitch_dict = {}
for k in pitch_annotations.keys():
  old_k = k
  if k == "Big rect. right mai":
      k = "Big rect. right main"
  elif k == "Big rect.  left bottom":
      k = "Big rect. left bottom"
  elif k == "Goal left post left ":
      k = "Goal left post left"
  corrected_pitch_dict[k] = pitch_annotations[old_k]

[Lehsuby]

Hello, @fmagera,

I applied all of your recommendations, but the results did not improve and remain far from the numbers reported in the paper.

Current results

{
  "completeness": 1.0,
  "meanRecall": 0.46686113,
  "meanPrecision": 0.1094238,
  "meanAccuracies": 0.10789514,
  "finalScore": 0.1078951433300972
}

What I’m attaching

• Tripod values for each test match (JSON).
• Camera parameters for each test video (JSON).

broadtrack_results_filter_players.zip

Could you please provide

• The exact evaluation script you used to compute JaC@5.
• A step-by-step plan describing precisely how the metrics in the paper were computed (preprocessing → detection → optical flow → player filtering → camera/tripod estimation → metric computation), including parameter values.
• Your recommended procedure to remove outliers (, https://github.com/user-attachments/assets/94b02d53-ce31-447c-8a75-d61e8b80f4b5).

Thank you!

[juliantziegler]

Hi @Lehsuby,

are there any new developments regarding your reproduction of the results?
I would be very interested, as I would like to benchmark my own method building upon NBJW against broadtrack.

@fmagera is there an implementation of broadtrack that you are aware of for the tracklab based sn-gamestate pipeline, that would help a lot.

Thanks!