PyRanges read_bed produces wrong number of chromosomes when cast to categorical

[mtekman]

First referenced here: sims-lab/CapCruncher#234 (comment)

Create Genomic Data

import pandas as pd
import numpy as np
## Create a dataframe with large number of rows
n = 150000
outtab = pd.DataFrame({'Chromosome': ([9] * (n-10)) + (['X'] * 10)}
    ).join(pd.DataFrame(np.random.randint(0,100, size=(n, 2)), columns=list(['Start','End']))
    ).join(pd.DataFrame({"Name" : ["Feat" + str(i) for i in range(n)] }))

## Write a file with headers, and one without
outtab.to_csv("/tmp/withheaders.bed", sep="\t", index=None)
outtab.to_csv("/tmp/withoutheaders.bed", sep="\t", index=None, header=None)

Produces a table with n rows, containing only two chromosomes: 9 and X

| Chromosome | Start | End | Name       |
|------------+-------+-----+------------|
| 9          |    43 |  72 | Feat0      |
| 9          |    20 |  12 | Feat1      |
| 9          |    37 |  60 | Feat2      |
| 9          |    85 |  48 | Feat3      |
| 9          |    54 |  96 | Feat4      |
| ...        |   ... | ... | ...        |
| X          |    94 |  40 | Feat149995 |
| X          |    29 |  11 | Feat149996 |
| X          |    96 |  39 | Feat149997 |
| X          |    92 |  36 | Feat149998 |
| X          |    44 |  54 | Feat149999 |

Read Bed File with Header

import pyranges as pr
tab = pr.read_bed("/tmp/withheaders.bed", as_df=True)    
## (No problems, it knows automatically to cast "Chromosome": "category")
## - Which chromosomes detected?
set(tab["Chromosome"].values)
## Produces: {'X', '9'}

Read Bed File without Header

import pyranges as pr
tab = pr.read_bed("/tmp/withoutheaders.bed", as_df=True)
## (Gives warning about mixed types, file too long for it to check)
## - Which chromosomes detected?
set(tab["Chromosome"].values)
## Produces: {9, 'X', '9'}
## --
## Cast chrom to categorical:
set(tab["Chromosome"].astype("category").values)  
## *Still* produces:  {'9', 9, 'X'}
## --
## Cast chrom to string first and then categorical
set(tab["Chromosome"].astype("string").astype("category").values) 
## Finally produces:  {'9', 'X'}

I don't know if this is expected behaviour or not when reading in a file without headers, but it's a pitfall I've seen.

Maybe it would be good if read_bed set the column names first, and then read the file?

[endrebak]

Thank you for reporting this. We have something in the works that will fix all of this