Parallel Python
Is your Python knowledge up to date? Shop for Python books on Amazon.
Parallel Python Community Forums rss  
November 29, 2015, 06:34:02 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Parallel python forum is up and running!
   Home   Help Search Login Register  
Pages: [1]
Author Topic: Remote ppserver setup script  (Read 7121 times)
0 Members and 1 Guest are viewing this topic.
« on: January 09, 2010, 05:22:49 PM »

Hi all,

I've attached two scripts that can be used to set up a number of ppserver(s) on remote hosts which will then be addressed by a local server with 0 workers. logs into the remote hosts using ssh via the paramiko module, checks the cpus online there as well as the current load and then sets up a ppserver with a number of workers accordingly.

You should at least start the script interactively so that you can type in a password in the shell if needed. You can however specify a logfile and simply send it to the background after entering the password and then disown it if you need to run this on a remote machine as well.

Also note that in line 549 the call to CLAMVParser is something rather internal. I changed the CLAMVParser because I did not want to reveal how to use our university's resources Tongue but it's a skeleton now for how to get a host list by parsing a url, maybe it's useful for someone.

Also I'd like to use threading for the part in lines 581-597 so that runs in parallel instead of in series. This may require changes to the ServerSetup class though.

Code: (python)
        for host in hosts:
           setup = ServerSetup(handler=handler, username=username, host=host,\
               port=ssh_port, buf_size=buf_size, timeout=timeout,\
               pp_port=pp_port, password=password, auto_add=auto_add,\
               workers =
           except IOError:
               logger.error("Communication for '%s@%s' failed!", username,
               count += workers
     "%d ppserver(s) now running on '%s'.", workers,\
               if not workers:

The script is meant to be changed for each job that you require. It should be in the same directory (or in PYHONPATH) as as it is imported by the latter. It's a simple interface to define everything the server needs to submit its tasks. If your callback function requires more arguments they should be put before server, result in the function definition.

I have absolutely not tested this script for Windows at all, if anyone ever needs that I'm happy for any hints/code snippets.

I also have not tested the --job-only or --servers-only options very much. I guess for --servers-only to work the remote ppservers would really have to be started with the -a option and you'd have to specify --hosts "*" for the script afterwards or so.

Also, please be patient for the jobs that you specify to be worked on. If you have many jobs it takes a while to submit all of them, plus parallel python seems to need a while to communicate with all remote ppservers. So just wait when nothing happens.

Enjoy, and comment Smiley

* (0.28 KB - downloaded 105 times.)
* (22.89 KB - downloaded 99 times.)
Pages: [1]
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.16 | SMF © 2011, Simple Machines Valid XHTML 1.0! Valid CSS!
Nutrition facts and analysis