python - n_jobs don't work in sklearn-classes -
does use "n_jobs" of sklearn-classes? work sklearn in anaconda 3.4 64 bit. spyder version 2.3.8. script can't finish execution after setting "n_jobs" parameter of sklearn-class non-zero value.why happening?
several scikit-learn tools such gridsearchcv , cross_val_score rely internally on python’s multiprocessing module parallelize execution onto several python processes passing n_jobs > 1 argument.
taken sklearn documentation:
the problem python multiprocessing fork system call without following exec system call performance reasons. many libraries (some versions of) accelerate / veclib under osx, (some versions of) mkl, openmp runtime of gcc, nvidia’s cuda (and many others), manage own internal thread pool. upon call fork, thread pool state in child process corrupted: thread pool believes has many threads while main thread state has been forked. possible change libraries make them detect when fork happens , reinitialize thread pool in case: did openblas (merged upstream in master since 0.2.10) , contributed patch gcc’s openmp runtime (not yet reviewed).
Comments
Post a Comment