Hi all,
I've attached two scripts that can be used to set up a number of ppserver(s) on remote hosts which will then be addressed by a local server with 0 workers. pp_setup.py logs into the remote hosts using ssh via the paramiko module, checks the cpus online there as well as the current load and then sets up a ppserver with a number of workers accordingly.
You should at least start the pp_setup.py script interactively so that you can type in a password in the shell if needed. You can however specify a logfile and simply send it to the background after entering the password and then disown it if you need to run this on a remote machine as well.
Also note that in line 549 the call to CLAMVParser is something rather internal. I changed the CLAMVParser because I did not want to reveal how to use our university's resources
but it's a skeleton now for how to get a host list by parsing a url, maybe it's useful for someone.
Also I'd like to use threading for the part in lines 581-597 so that runs in parallel instead of in series. This may require changes to the ServerSetup class though.
for host in hosts:
setup = ServerSetup(handler=handler, username=username, host=host,\
port=ssh_port, buf_size=buf_size, timeout=timeout,\
pp_port=pp_port, password=password, auto_add=auto_add,\
pp_secret=pp_secret)
try:
workers = setup.run()
except IOError:
logger.error("Communication for '%s@%s' failed!", username,
host)
rm.append(host)
else:
count += workers
logger.info("%d ppserver(s) now running on '%s'.", workers,\
host)
if not workers:
rm.append(host)
The pp_task.py script is meant to be changed for each job that you require. It should be in the same directory (or in PYHONPATH) as pp_setup.py as it is imported by the latter. It's a simple interface to define everything the server needs to submit its tasks. If your callback function requires more arguments they should be put before server, result in the function definition.
I have absolutely not tested this script for Windows at all, if anyone ever needs that I'm happy for any hints/code snippets.
I also have not tested the --job-only or --servers-only options very much. I guess for --servers-only to work the remote ppservers would really have to be started with the -a option and you'd have to specify --hosts "*" for the pp_setup.py script afterwards or so.
Also, please be patient for the jobs that you specify to be worked on. If you have many jobs it takes a while to submit all of them, plus parallel python seems to need a while to communicate with all remote ppservers. So just wait when nothing happens.
Enjoy, and comment
Forums 
