Add UTF-8 support in data (lookup tables)

[reinholdsson]

Need to find a solution for dealing with åäö. See examples below:

library(Coldbir)
a <- cdb()
dt <- data.table(
    x = c('a', 'b', 'a', 'o', 'a', 'o', 'o'),
    y = c('a', 'b', 'å', 'ö', 'a', 'ö', 'ö')
)
a[] <- dt
# Warning message:
# In `[.data.table`(y, xkey, nomatch = ifelse(all.x, NA, 0), allow.cartesian = allow.cartesian) :
#   A known encoding (latin1 or UTF-8) was detected in a join column. data.table compares the bytes 
# currently, so doesn't support *mixed* encodings well; i.e., using both latin1 and UTF-8, or if any 
# unknown encodings are non-ascii and some of those are marked known and others not. But if either 
# latin1 or UTF-8 is used exclusively, and all unknown encodings are ascii, then the result should be ok. 
# In future we will check for you and avoid this warning if everything is ok. The tricky part is doing this 
# without impacting performance for ascii-only cases.

a[]
#    x y
#1: a a
#2: b b
#3: a  
#4: o  
#5: a a
#6: o  
#7: o  
# Warning message:
# In `levels<-`(`*tmp*`, value = c("a", "b", "", "")) :
#   duplicated levels in factors are deprecated

lookup.txt for variable y:

1       a
2       b
3
4

[reinholdsson]

The first warning is due to this part:

# Convert variable (TODO: rewrite this part)
if (!is.null(lookup)) {
  setnames(lookup, c("k", "v"))
  x_data <- data.table(k = as.integer(x), v = x, order = 1:length(x))
  x <- merge(x_data, lookup, by = "v", all.x = T, all.y = F)
  x <- x$k.y[order(x$order)]
}

However, this might bot be critical, as long as the vector has the same encoding (e.g., not mixing latin1 with utf-8).

While the second is due escape_char. To avoid empty values in lookup table (or repeating ones), one solution is to move the escape function onto the variable instead of the lookup table. Also, it might be an improvement to change it to gsub("\\t|\\n|\\a|\\f|\\r", "", x) to allow åäö. At least until we find a better way to escape or quote characters.

Update: Moving the escape_char function to run on the variable decreases the performance way too much ...