r - Saving multiple outputs of foreach dopar loop

Question

I would like to know if/how it would be possible to return multiple outputs as part of foreach dopar loop.

Let's take a very simplistic example. Let's suppose I would like to do 2 operations as part of the foreach loop, and would like to return or save the results of both operations for each value of i.

For only one output to return, it would be as simple as:

library(foreach)
library(doParallel)
cl <- makeCluster(3)
registerDoParallel(cl)

oper1 <- foreach(i=1:100000) %dopar% {
    i+2
}

oper1 would be a list with 100000 elements, each element is the result of the operation i+2 for each value of i.

Suppose now I would like to return or save the results of two different operations separately, e.g. i+2 and i+3. I tried the following:

oper1 = list()
oper2 <- foreach(i=1:100000) %dopar% {
    oper1[[i]] = i+2
    return(i+3)
}

hoping that the results of i+2 will be saved in the list oper1, and that the results of the second operation i+3 will be returned by foreach. However, nothing gets populated in the list oper1! In this case, only the result of i+3 gets returned from the loop.

Is there any way of returning or saving both outputs in two separate lists?

Answer 1

Don't try to use side-effects with foreach or any other parallel program package. Instead, return all of the values from the body of the foreach loop in a list. If you want your final result to be a list of two lists rather than a list of 100,000 lists, then specify a combine function that transposes the results:

comb <- function(x, ...) {
  lapply(seq_along(x),
    function(i) c(x[[i]], lapply(list(...), function(y) y[[i]])))
}

oper <- foreach(i=1:10, .combine='comb', .multicombine=TRUE,
                .init=list(list(), list())) %dopar% {
  list(i+2, i+3)
}

oper1 <- oper[[1]]
oper2 <- oper[[2]]

Note that this combine function requires the use of the .init argument to set the value of x for the first invocation of the combine function.