Multi-Processing in Python

Python has built-in multiprocessing module in its standard library.

One simple example of using the Pool class is the following.

def myfunc(myfuncargs):
    'some thing here'

with Pool(10) as p:
    records = p.map(myfunc, myfuncargs)

However, there are limitations on this, especially on pickles. Another approach.

from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

with ThreadPool(1) as p:
     records = p.map(myfunc, myfuncargs)

Beware that map function will feed in a list of args to the function. So I have to use p.map(myfunc, [arg]) for one arg.


Back to top

© 2016-2018, Lei Ma | Created with Sphinx and . | On GitHub | Physics Notebook Statistical Mechanics Notebook Neutrino Physics Notes Intelligence | Index | Page Source