However, if you're primarily interacting with other systems (running HTTP commands, talking with a database, writing to filesystems) then your code is likely not bound by CPU and won't take much of a hit. The downside to this method is that python isn't the greatest language with handling threads- it uses something called the Global Interpreter Lock to stay thread safe, which can slow down some use cases here. This works because ThreadPool shares memory with the main thread, rather than creating a new process- this means that pickling is not required. This can be done with no change of code other than the import- from multiprocessing.pool import ThreadPool as Pool When this problem comes up with multiprocessing a simple solution is to switch from Pool to ThreadPool. Job = apply_async(pool, lambda a, b: (a, b, a * b), (i, i + 1)) Return pool.apply_async(run_dill_encoded, (payload,)) This solution requires only the installation of dill and no other libraries as pathos: import os If you cannot reorganize your code as described by unutbu, you can use dills extended pickling/unpickling capabilities for transferring data (especially code data) as I show below.
Notice that foo is pickable, since Foo is defined at the top level and foo._dict_ is picklable.Īs others have said multiprocessing can only transfer Python objects to worker processes which can be pickled. It can be fixed by defining a function at the top level, which calls foo.work(): def work(foo): Everything that goes through the mp.SimpleQueue must be pickable, and foo.work is not picklable since it is not defined at the top level of the module. The problem is that the pool methods all use a mp.SimpleQueue to pass tasks to the worker processes. Yields an error almost identical to the one you posted: Exception in thread Thread-2:įile "/usr/lib/python2.7/threading.py", line 552, in _bootstrap_innerįile "/usr/lib/python2.7/threading.py", line 505, in runįile "/usr/lib/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks This piece of code: import multiprocessing as mp In particular, functions are only picklable if they are defined at the top-level of a module. I tried simpler example with this pattern and it works though. f(), g(), h() are all defined at the top level. i.e, f() calls g() calls h() which has a nested function i(), and I am calling pool.apply_async(f). Though it calls a function that contains a nested function. Update: The function I pickle is defined at the top level of the module. PicklingError: Can't pickle : attribute lookup _builtin_.function failed Self._target(*self._args, **self._kwargs)įile "/usr/lib64/python2.7/multiprocessing/pool.py", line 313, in _handle_tasks
Exception in thread Thread-3:įile "/usr/lib64/python2.7/threading.py", line 552, in _bootstrap_innerįile "/usr/lib64/python2.7/threading.py", line 505, in run They were all caused by using pool to call function defined within a class function. I looked up some previous notes on this problem. If I run the program in IPython shell instead of the regular Python, things work out well. I am sorry that I can't reproduce the error with a simpler example, and my code is too complicated to post.
0 kommentar(er)
