Where: faircode/profiler.py's _looks_like_dates (age-vs-date column detection), interacting with _age_to_numeric.
The gap: _looks_like_dates samples series.dropna().astype(str).head(50) - literally the first 50 rows, not a representative sample despite what "looks like" implies. If a column has clean numeric ages in the first 50 rows and date strings appended later (e.g. a second data source merged in below the original rows), the date-check passes on the sampled head, so the whole column is treated as numeric age - and _age_to_numeric's re.search(r"\d+", ...) then grabs the leading digit run of each date string (e.g. "1985-03-21" -> 1985), banding it straight into "75+".
Repro:
>>> import pandas as pd
>>> from faircode import profile
>>> ages = list(range(20, 80))
>>> dates = ['1985-03-21', '1990-07-14', '2001-11-02'] * 20
>>> df = pd.DataFrame({'age': ages + dates})
>>> for g in profile(df)['dimensions'][0]['groups']: print(g['label'], g['count'])
75+ 65
30-45 15
45-60 15
60-75 15
18-30 1060 birthdate strings are silently collapsed into the oldest age band, with no flag anywhere.
Why it matters: this is exactly the failure mode _looks_like_dates exists to prevent (see SPEC section 2) - defeated by sampling only the head of the column instead of throughout it. Produces a fabricated elderly-skew from what's actually garbage/mixed-source input.
Suggested fix: sample throughout the column instead of .head(50) - e.g. series.dropna().astype(str).sample(min(200, len(series)), random_state=0), or every Nth row - so a column with dates appended after the initial rows is still correctly detected.
Where:
faircode/profiler.py's_looks_like_dates(age-vs-date column detection), interacting with_age_to_numeric.The gap:
_looks_like_datessamplesseries.dropna().astype(str).head(50)- literally the first 50 rows, not a representative sample despite what "looks like" implies. If a column has clean numeric ages in the first 50 rows and date strings appended later (e.g. a second data source merged in below the original rows), the date-check passes on the sampled head, so the whole column is treated as numeric age - and_age_to_numeric'sre.search(r"\d+", ...)then grabs the leading digit run of each date string (e.g."1985-03-21"->1985), banding it straight into"75+".Repro:
60 birthdate strings are silently collapsed into the oldest age band, with no flag anywhere.
Why it matters: this is exactly the failure mode
_looks_like_datesexists to prevent (see SPEC section 2) - defeated by sampling only the head of the column instead of throughout it. Produces a fabricated elderly-skew from what's actually garbage/mixed-source input.Suggested fix: sample throughout the column instead of
.head(50)- e.g.series.dropna().astype(str).sample(min(200, len(series)), random_state=0), or every Nth row - so a column with dates appended after the initial rows is still correctly detected.