foreach - Parallel processing in R done wrong? -
i have code trying process in parallel using foreach-package. code working when run on computer 4 cores takes 26 min , when switch 1 32 cores, still takes 13 min finish. wondering whether doing wrong since using 8 times cores, reduce time 1 half. code looks this:
no_cores <- detectcores() cl <- makecluster(no_cores) registerdoparallel(cl) xenopus_data <- foreach(b=1:length(newly_populated_vec),.packages = c("raster", "gdistance", "rgdal","sp")) %dopar% { xenopus_walk(altdata=altdata,water=water,habitat_suitability=habitat_suitability,max_range_without_water=max_range_without_water,max_range=max_range,slope=slope,start_pt=newly_populated_vec[b]) } stopcluster(cl)
for computer 4 cores following time:
time_of_start [1] "2016-07-12 13:07:23 cest" time_of_end [1] "2016-07-12 13:33:10 cest" , 1 32 cores: time_of_start [1] "2016-07-12 14:35:48 cest" time_of_end [1] "2016-07-12 14:48:08 cest"
is normal ? , if so, know how speed additionally, maybe using different packages? type of appreciated!
edit: these times after applying corrections suggested. 32 cores:
user system elapsed 5.99 40.78 243.97
for 4 cores:
user system elapsed 1.91 0.94 991.71
note before, did calculation multiple times via loops, that's why computation time decreased drastically, 1 can still tell difference between 2 computers has increased, believe.
try , let me know if problem solved:
library(doparallel) library(foreach) registerdoparallel(cores=detectcores()) n <- length(newly_populated_vec) cat("\nn = ", n, " | parallel workers count = ", getdoparworkers(), "\n\n", sep="") t0 <- proc.time() xenopus_data <- foreach(b=1:n,.packages = c("raster", "gdistance", "rgdal","sp"), .combine=rbind) %dopar% { xenopus_walk( water=water, altdata=altdata, habitat_suitability=habitat_suitability, max_range_without_water=max_range_without_water, max_range=max_range, slope=slope, start_pt=newly_populated_vec[b]) } time <- proc.time() - t0
also, try monitor logical cores in pc/laptop check if cores involved in computation. (taskmanager windows , htop
linux)
please mindful doubling number of cores not lead having double performance.
Comments
Post a Comment