Handle Ambiguous Scene Descriptions in Synthetic Evaluation

[creativeessence]

Problem

Synthetic queries like:

“2 characters fight each other with guns”

may match multiple scenes in a video. Miners may return a different valid fight scene than the validator’s chosen ground-truth clip, causing unfair penalties.

Why It Matters

This directly affects:

• Miner fairness
• Validator scoring integrity
• Incentive alignment on Bittensor

Proposed Direction

• Enforce temporal grounding via additional constraints
• Use scene disambiguation hints (e.g., clothing, location)
• Penalize only if semantic mismatch > threshold
• Allow multiple valid ground-truth intervals

Acceptance Criteria

• Validator scoring does not unfairly penalize semantically correct alternative scenes.
• Evaluation metrics robust to multi-instance events.

[IlyasFardaouix]

I think the root cause here lies in the fact that the scene disambiguation logic is currently not considering temporal grounding constraints, which could lead to multiple valid fight scenes being matched by the miner. Have you explored adding additional constraints to the scene matching algorithm to enforce temporal grounding, as proposed in the direction? Specifically, I'd like to know if you've considered how to incorporate scene disambiguation hints (e.g., clothing, location) into the existing implementation.