Parallel Python Community Forums

Python Forums => Parallel Python Forum => Topic started by: Vitalii on June 11, 2008, 10:17:05 AM

Title: FAQ, Parallel Python
Post by: Vitalii on June 11, 2008, 10:17:05 AM
Q) How to check that execution of a task has been completed?

A) Task object, returned by submit call, has an argument finished which indicates status of the task:
Code: (python)
task = job_server.submit(f1, (a,b,c))

Code: (python)
if task.finished:
   print "The task is done!"
   print "Still working on it ..."

Q) I would like to perform an action at the time of completion for each individual task. Is it possible?

A) Yes, you need to setup callback argument of submit method, see PP callback example (

Q) Is it possible to use Psyco with PP?

A) Yes it is definitely possible. This post  (,com_smf/Itemid,29/topic,185.msg583#msg583) explains how to incorporate Psyco in PP.

Q) I do not want to run any tasks on my local computer. How do I do that?

A) ncpus argument of the pp.Server specifies the number of local ppworkers to use. If you do not specify this argument PP uses the numbers of ppworkers equal to the number of effective processors on your local host. If you set it to 0, three will be no local execution of tasks.

Q) How can I use pp.Template class in my program?

A) The purpose of Template class is to simplify submission of similar pp-tasks, which execute the same function with different arguments. There are two steps in using the Template class.
1) Creating a template which glues together a function with its dependencies and given instance of pp.Server:
Code: (python)
fn = pp.Template(job_server, sum_primes, (isprime,), ("math",))
2) Using the template to submit jobs with different arguments:
Code: (python)
job1 = fn.submit(100)
job2 = fn.submit(200)
Please look at the following example (,com_smf/Itemid,29/topic,234.msg703#msg703) for further clarification.

Q) How to use PP on a PBS cluster?

A) PP can be used with PBS with or without autodiscovery.
Without autodiscovery you simply need to start on all nodes assigned to a job and then add all of them to a ppserver variable in the client Python application.
With autodiscovery you need to run with -a switch (./ -a ) and then set ppservers = ("*",) in the client application.
In both cases it is useful to use -t switch ( to automatically kill all ppservers when client exits.

Q) What Python versions are supported?

A) PP was tested with Python 2.3 - 2.7 on a variety of platforms.
Python 3 port is available ( as well.

Q) How load balancing works in Parallel Python?

A) PP uses dynamic load balancing. Both local server and remote ppservers use ppworkers to execute jobs one at a time. Initially jobs are sent to remote ppservers according to the number of ppworkers they have. The number of remote jobs exceeds the number of ppworkers so that job input & output transfer could be interleaved with the computation.  When execution of a remote job finishes, another job will be scheduled. As a result each host executes the number of jobs proportional to their computational capacity.
When both local and remote hosts are present, jobs are first scheduled on the localhost.

Q) What to do if large jobs execute correctly locally but I get "Socket connection is broken" when the run on remote ppserver?

A) Increase the socket timeout time (socket_timeout argument of the Server and -k command line argument of to be an upper bound of your job execution time.

Q) Can Parallel Python work across different versions of Python?

A) Yes, the same version of Parallel Python works across different versions of Python as long as all of them are either Python 2.x or Python 3.x. For example Parallel Python will works on cluster that has Python 2.5, 2.6, 2.7 installed on various machines. However it won't work if ppclient is using Python 2.7 and ppserver is using Python 3.5.

(to be continued)
Please feel free to suggest question and answers for this FAQ in replies to the post.

Title: Re: FAQ, Parallel Python
Post by: wkerzendorf on April 13, 2010, 11:33:24 PM
Q: I get a Socket Error/Memory Error when using jobs that use os.system calls

A: The fix I found is using subprocess.Popen and poping the stdout,stderr into the subprocess.PIPE. here is an example:
subprocess.Popen(['ls -rtl'],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True). That fixed the error for me.