Skip to content

graph_node_classification: a complete integer-keyed answer scores about 2 #46

Closed as not planned

Description

@findalexli

Task

graph_node_classification

Claim

problem.md says prediction keys "can be strings or values convertible to strings", and test_nodes arrives as integers. The verifier looks up string IDs only, so a complete answer keyed by integers matches no node. Every quality metric is 0, and the run is still valid because the coverage check only counts keys.

Impact

The same predictions from an Opus 5.5 solution score 90.54-90.63 with string keys and 1.95-2.01 with integer keys (n=4). We know of no natural case: the starter and 105 solutions we hold all key by str(nid).

Evidence and how to reproduce

Change a solution's output keys from str(nid) to nid, and grade both with tools/judge_local.py. overall goes from 0.885740 to 0.

Proposed fix

Draft PR: #37.

Activity

  1. findalexli commented on Nov 6, 2026

    @findalexli
    ContributorAuthor

    Closing with #37, for the same reason: no real submission we checked uses integer keys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions