You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
graph_node_classification: a complete integer-keyed answer scores about 2 #46
problem.md says prediction keys "can be strings or values convertible to strings", and test_nodes arrives as integers. The verifier looks up string IDs only, so a complete answer keyed by integers matches no node. Every quality metric is 0, and the run is still valid because the coverage check only counts keys.
Impact
The same predictions from an Opus 5.5 solution score 90.54-90.63 with string keys and 1.95-2.01 with integer keys (n=4). We know of no natural case: the starter and 105 solutions we hold all key by str(nid).
Evidence and how to reproduce
Change a solution's output keys from str(nid) to nid, and grade both with tools/judge_local.py. overall goes from 0.885740 to 0.
Activity
findalexli commentedon Nov 6, 2026
Closing with #37, for the same reason: no real submission we checked uses integer keys.