ubuntu - Parallel processing in R limited

Question

I'm running ubuntu 12.04 and R 2.15.1 with the parallel and doParallel packages. when I run anything in parallel I'm limited to 100% of a core, when I should have up to 800%, since I am running it with 8 cores. What shows up on the system monitor is that each child process is getting only 12%.

What is going on that is limiting my execution speed?

Answer 1

The problem may be that the R process is restricted to one core (and the subprocesses inherit that).

Try this:

> system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))
pid 3064's current affinity mask: fff
pid 3064's new affinity mask: fff

Now, if on your machine, the current affinity mask reports a 1, then this was the problem. The line above should solve it (i.e. the second line should report fff (or similar).

Simon Urbanek wrote a function mcaffinity that allows this control for multicore. As far as I know, it's still in R-devel.

For details, see e.g. this discussion on R-sig-hpc.

Update, and addition to Xin Guo's answer:

If you use implicit parallelization via openblas and explicit parallelization (via parallel/snow/multicore) together, you may want to change the number of threads that openblas uses depending on whether you are inside an explicitly parallel part or not.
This is possible (with openblas under Linux, I'm not aware of any other of the usual optimized BLAS' that provides a function to the number of threads), see Simon Fuller's blog post for details.