r - How to split dataframe for future_map for optimal performance

Question

I am running a data preparation script which involves a step in which the rows of the dataframe are split up into multiple observations. This step takes quite some time and I'm trying to optimize the use of future to get some improvements. The dataframe that I'm working on is very large (about 3 million rows) and I'm trying to find the optimal way of splitting it up in chunks to allow for parallel processing. I'm running this on 64 cores so there is potential for speeding up, but I'm unsure how not to run into memory problems or where I loose my speed.

I ran some small experiments and got quite confused by the results. I experimented with nested future maps, which I thought should not increase performance as the second level of futures runs sequential. But it does speed up things.

library(tidyverse)
library(furrr)
library(tictoc)

flights <- nycflights13::flights

create_rows <- function(data, split_by, sep_by){
  split_by <- enquo(split_by)
  sep_by <- enquo(sep_by)
  
  data %>% 
    group_split(!!split_by) %>% 
    future_map_dfr(~.x %>% uncount(weights = !!sep_by))
}

availableCores()
#system 
#    64
future::plan(multisession, workers = availableCores() -1 )

tic()
long_df <- flights %>% 
  create_rows(data = ., split_by = day, sep_by = month)
toc()
#41.14 sec elapsed

tic()
long_df2 <- flights %>%
  group_split(day) %>% 
  future_map_dfr(~create_rows(data = ., split_by = carrier, sep_by = month))
toc()
#15.349 sec elapsed

# Less workers
future::plan(multisession, workers = 8)
tic()
long_df <- flights %>% 
  create_rows(data = ., split_by = day, sep_by = month)
toc()
#12.239 sec elapsed

tic()
long_df2 <- flights %>%
  group_split(day) %>% 
  future_map_dfr(~create_rows(data = ., split_by = carrier, sep_by = month))
toc()
#6.61 sec elapsed

# Different splits

tic()
long_df <- flights %>% 
  create_rows(data = ., split_by = carrier, sep_by = month)
toc()
#11.278 sec elapsed
tic()
long_df2 <- flights %>%
  group_split(carrier) %>% 
  future_map_dfr(~create_rows(data = ., split_by = day, sep_by = month))
toc()
#5.235 sec elapsed

Is it generally advisable to split up the dataframe in as small bits as possible and assign them to different processes or is there an optimal amount that gets passed to each? I seem to get improvements in speed when reducing the amount of workers. Should I reduce the split of the dataframe to match the amount of workers?

I'm generally quite confused right now, and would be very grateful for helpful explanations.

question from:https://stackoverflow.com/questions/65846252/how-to-split-dataframe-for-future-map-for-optimal-performance

Answer 1

Waitting for answers