Parallel version of `[[`

[reinholdsson]

Although the solution below isn't faster in parallel on my personal computer, it's at least an example on how to read in parallel:

library(foreach)
library(doParallel)
library(Coldbir)
library(data.table)

# Prepare data
N <- 10000000L

x <- data.table(
  a = as.Date(sample(c(1000+1:100), N, replace = T), origin = "1960-01-01"),
  b = sample(1:100, N, replace = T),
  c = sample(1:10000, N, replace = T),
  d = sample(LETTERS, N, replace = T)
)

a <- cdb()
a[] <- x

cols <- names(x)

# Read non-parallel
system.time({res <- a[]})

# Read parallel (by column)
cl <- makeCluster(4)
registerDoParallel(cl)

system.time({
  res <- as.data.table(foreach(i = cols) %dopar% {
    require(Coldbir)
    a[i][[1]]
  })
  setnames(res, cols)
})

stopCluster(cl)

Perhaps we can figure out a better use case.

[mikula]

Nice test, although little bit depressing. It is too much to combine from each iteration. Parallel computation is powerful for independent data sets reduced into small output-data.