Where: faircode/profiler.py's profile() (if dimensions: overall = ... else: overall = 0), and faircode/report.py's to_terminal/to_json.
The gap: a dataset with zero detected/kept dimensions (e.g. every demographic column force-mapped to ignore, or none present at all) gets overall_score=0, grade="F" - numerically and visually identical to a dataset that genuinely failed every fairness check. report.py's "No demographic columns detected." note exists only in to_terminal (line 55); it's never surfaced in the raw profile() dict or to_json's output.
Repro:
>>> import pandas as pd
>>> from faircode import profile
>>> df = pd.DataFrame({'sex': ['M','F']*20, 'id': range(40)})
>>> r = profile(df, overrides={'sex': 'ignore'})
>>> r['dimensions']
[]
>>> r['overall_score'], r['grade']
(0, 'F')Nothing in this dict distinguishes "nothing was measured" from "everything failed."
Why it matters: this same profile() backs the MCP profile_dataset/compare_datasets tools and faircode profile --json/--fail-under. An agent or a CI gate sees {"overall_score": 0, "grade": "F"} and reasonably concludes severe bias, when actually nothing was assessed - a --fail-under gate would hard-fail for entirely the wrong reason, and an MCP-calling agent has no field to check for "was anything even measured here."
Suggested fix: use a distinct sentinel for the zero-dimensions case (e.g. overall_score: null/grade: None plus a "dimensions_detected": false field, or similar) instead of reusing the worst possible numeric score and letter grade, and propagate the existing terminal-only "no demographic columns detected" note into to_json's output and the MCP tool results too.
Where:
faircode/profiler.py'sprofile()(if dimensions: overall = ... else: overall = 0), andfaircode/report.py'sto_terminal/to_json.The gap: a dataset with zero detected/kept dimensions (e.g. every demographic column force-mapped to
ignore, or none present at all) getsoverall_score=0, grade="F"- numerically and visually identical to a dataset that genuinely failed every fairness check.report.py's"No demographic columns detected."note exists only into_terminal(line 55); it's never surfaced in the rawprofile()dict orto_json's output.Repro:
Nothing in this dict distinguishes "nothing was measured" from "everything failed."
Why it matters: this same
profile()backs the MCPprofile_dataset/compare_datasetstools andfaircode profile --json/--fail-under. An agent or a CI gate sees{"overall_score": 0, "grade": "F"}and reasonably concludes severe bias, when actually nothing was assessed - a--fail-undergate would hard-fail for entirely the wrong reason, and an MCP-calling agent has no field to check for "was anything even measured here."Suggested fix: use a distinct sentinel for the zero-dimensions case (e.g.
overall_score: null/grade: Noneplus a"dimensions_detected": falsefield, or similar) instead of reusing the worst possible numeric score and letter grade, and propagate the existing terminal-only "no demographic columns detected" note intoto_json's output and the MCP tool results too.