Where: assets/profiler-engine.js's sniffDelimiter (splits raw text via .split(/\r\n|\r|\n/), not quote-aware) vs faircode/loaders.py's _sniff_delimiter (uses Python's csv.Sniffer, which is quote-aware).
The gap: for a delimited file with an unrecognized extension (so both engines take the sniffing code path - see the companion .csv-extension issue for the recognized-extension case), if one of the first sampled lines is a quoted field containing a literal newline, JS's naive line-split corrupts the sample: the embedded newline creates a spurious extra "line" mid-field, so the per-line delimiter counts stop being consistent and it falls back to the hardcoded default , - wrong. Python's csv.Sniffer still gets the real delimiter right on the identical file, since it understands quoting.
Repro: a file sniff_quoted_newline.dat (unrecognized extension), ;-delimited, with a quoted multi-line field in one row:
sex;race;age;notes
M;White;25;"single line"
F;Black;30;"multi
line note"
M;White;45;"ok"
F;Asian;22;"fine"
M;White;50;"good"
>>> from faircode.loaders import read_table
>>> read_table('sniff_quoted_newline.dat').shape
(5, 4)Correct - 5 rows, 4 columns. Through the JS engine:
> E.sniffDelimiter(fs.readFileSync('sniff_quoted_newline.dat','utf8'))
','
> E.parseCSV(...).columns
[ 'sex;race;age;notes' ]Sniffs the wrong delimiter and collapses into one garbage column.
Why it matters: same shared "sniff" logic, identical file, opposite outcomes - a real fairness dimension collapses into one bogus column in the browser tool while the CLI reads it correctly.
Suggested fix: make sniffDelimiter's line-sampling quote-aware - e.g. reuse parseCSV's own row-tokenizer to split rows before sampling for delimiter frequency, instead of a naive regex split on \n/\r.
Where:
assets/profiler-engine.js'ssniffDelimiter(splits raw text via.split(/\r\n|\r|\n/), not quote-aware) vsfaircode/loaders.py's_sniff_delimiter(uses Python'scsv.Sniffer, which is quote-aware).The gap: for a delimited file with an unrecognized extension (so both engines take the sniffing code path - see the companion
.csv-extension issue for the recognized-extension case), if one of the first sampled lines is a quoted field containing a literal newline, JS's naive line-split corrupts the sample: the embedded newline creates a spurious extra "line" mid-field, so the per-line delimiter counts stop being consistent and it falls back to the hardcoded default,- wrong. Python'scsv.Snifferstill gets the real delimiter right on the identical file, since it understands quoting.Repro: a file
sniff_quoted_newline.dat(unrecognized extension),;-delimited, with a quoted multi-line field in one row:Correct - 5 rows, 4 columns. Through the JS engine:
Sniffs the wrong delimiter and collapses into one garbage column.
Why it matters: same shared "sniff" logic, identical file, opposite outcomes - a real fairness dimension collapses into one bogus column in the browser tool while the CLI reads it correctly.
Suggested fix: make
sniffDelimiter's line-sampling quote-aware - e.g. reuseparseCSV's own row-tokenizer to split rows before sampling for delimiter frequency, instead of a naive regex split on\n/\r.