Skip to content

refactor: remove 14 hardcoded benchmark branches from query/agent.py (CODE_HEALTH #2) #39

Description

@duanyiqun

CODE_HEALTH #2 (see docs/CODE_HEALTH.md in #37).

src/cognifold/query/agent.py:789-808 hardcodes a dispatch dict of 14 benchmark names mapping to 14 private _query_<bench>_qa methods (lines 811–1261) — several differing only by a magic max_nodes (40/15/5/30). Six entries (msc, qmsum, rgb, socialiqa, safetybench, futurex) refer to benchmarks whose directories no longer exist. query() itself is 263 lines; MemoryQueryAgent has 44 methods.

Cost: the shipped library carries leaderboard tuning; adding a benchmark means editing the core query engine; library users get 14 dead branches.

Fix direction:

  • Introduce a QueryProfile dataclass (max_nodes, boosts, prompt template) accepted by query() / query_for_qa()
  • Move per-benchmark values into benchmarks/*/ or configs/<name>_profile.yaml
  • Delete the dispatch dict + 14 methods
  • Do this behind tests (#CODE_HEALTH-1) and verify with --limit 1 runs of locomo/musique/tomi

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions