Competition Ranking Calculation

[NONONOexe]

I would like to confirm an inconsistency I found regarding the calculation of competition rankings across the rules, the programs, and past competion results.

According to the 2025 competition rules, the ranking should be calculated as described in Section "3.3 Ranking" of this document. This procedure has remained unchanged since it was originally proposed in 2015 competition.

In this calculation, I found a probrem with the following formula:

$$ TP_{ji}^k = step \therefore MSS_{j/step}^k < SC_{ji}^k < MSS_{j / step+1}^k $$

Here, the variable $MSS_{i/step}^k$ is normalized according to the following:

$$ MSS_{j/step}^k = \frac{max(SC_j^k) - SM_j^k}{MS_j^k \times (MS_j^k - step)} $$

However, the variable $SC_{ji}^k$ represents a raw score, so they are no different scales which makes the calculation inconsistent.

When calculating MSS, its values increse exponentially from near zero as the step approaches $2 \times n$ ($n$ is number of teams). Therefore, I think the formula needs to be adjusted so that MSS-s can function properly as a step boundaries---for example, by substracting the minimum score from $SC_{ji}^k$ to use it as the baseline.

On the other hand, in the implementation (make_html.py), the calculation differs from the rules. This program has remained the same for about 12 years. Unlike the rules, where the step widths are defined using $MSS_{j/step}^k$, the program uses a fixed step width.

From the code, it can be confirmed that the implementation defines the step width as:

best = sorted_by_score[0].final_score # max score
n = len(entries) # number of teams
delta = best / (2.0*n) # step width (delta) is fixed

Looking at past competition points, this definition of delta is too large and does not match the published results. Instead, when I tried using:

delta = (max(scores) - ave(scores)) / n

the values matched almost exactly.

As a result, we now have three different approaches:

• The rules (exponentially increasing step width),
• The program (fixed step width based on best / (2.0*n), and
• The past results (witch seem to match with (max(scores) - ave(scores)) / n.

My question is which of these should be considered the correct method for calculating rankings in the RoboCupRescue Simulation competitions?

[gnardin]

Dear @NONONOexe
Find attached a spreadsheet we used in 2015 to evaluate the Rank Calculation as described in the Competition Rules document.
RCRS-Rank-Calculation-example.xls

The formal representation may be wrong, but the TP^k_{ji} returns the step value, but range tested are based on the raw value. Please see how this is calculated in the column C rows 23-26 in the spreadsheet. Therefore, I do not consider the formulas described in the rules are incorrect or have any scale issues.

But I do not have an answer to the discrepancy between the rules description and the actual calculation for the competitions. @ffaraji or @modaresimr may provide an explanation since they implemented the Python code for calculating the score during the competition and ran most of the competitions.

My answer to your question is to use the formula described in the Rules document.

Please let me know if you need any further clarification concerning the spreadsheet I shared.

Kind regards,

[NONONOexe]

Dear @gnardin

Thank you for your explanation.
I appreciate you sharing the information.
I'll review the spreadsheet you shared and take a closer look at the calculations.

Best regards,