Skip to content

faircode/SPEC.md's MCP tools table says get_benchmark_results reads paper/results-frozen/, contradicting the code and its own prose #434

Description

@yakew7

Where: faircode/SPEC.md section 11's MCP tools table, the get_benchmark_results row, vs faircode/mcp_server.py:68-72,296-298 and SPEC.md's own surrounding prose.

The gap: the table cell says get_benchmark_results wraps paper/results-frozen/results_{fairness,performance}.csv. The paragraph directly below the table, and the actual code, both say it reads the package-internal generated mirror instead:

Repro:

$ grep -n "results-frozen/results_" faircode/SPEC.md
376:| `get_benchmark_results` | `paper/results-frozen/results_{fairness,performance}.csv` | ...

$ grep -n "RESULTS_FROZEN_DIR" faircode/mcp_server.py
68:RESULTS_FROZEN_DIR = Path(__file__).resolve().parent / "_results_frozen"
70:    "fairness": RESULTS_FROZEN_DIR / "results_fairness.csv",
71:    "performance": RESULTS_FROZEN_DIR / "results_performance.csv",

RESULTS_FROZEN_DIR is faircode/_results_frozen/, not paper/results-frozen/.

Why it matters: SPEC.md is the load-bearing reference for exactly this distinction elsewhere in the same document (explaining why explainers/ and paper/results-frozen/ aren't files a real pip install faircode[mcp] ships, and why faircode/_explainers//faircode/_results_frozen/ exist as generated package-internal mirrors instead). The table cell directly contradicts that explanation, so a reader trusting the table gets the wrong mental model of where a real installed package actually reads from - the same category of gap #388 fixed for the missing mirrors themselves.

Suggested fix: change the table cell to faircode/_results_frozen/results_{fairness,performance}.csv (mirrored from paper/results-frozen/ via scripts/freeze_paper_results.mirror_for_mcp()).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomershelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions