Parallel Python
Parallel Python Community Forums rss  
October 31, 2014, 09:18:53 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Parallel python forum is up and running!
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: Remote ppserver setup script  (Read 4945 times)
0 Members and 1 Guest are viewing this topic.
Midnighter
Full Python
***

Karma: 0
Posts: 13


View Profile
« on: January 09, 2010, 05:22:49 PM »

Hi all,

I've attached two scripts that can be used to set up a number of ppserver(s) on remote hosts which will then be addressed by a local server with 0 workers. pp_setup.py logs into the remote hosts using ssh via the paramiko module, checks the cpus online there as well as the current load and then sets up a ppserver with a number of workers accordingly.

You should at least start the pp_setup.py script interactively so that you can type in a password in the shell if needed. You can however specify a logfile and simply send it to the background after entering the password and then disown it if you need to run this on a remote machine as well.

Also note that in line 549 the call to CLAMVParser is something rather internal. I changed the CLAMVParser because I did not want to reveal how to use our university's resources Tongue but it's a skeleton now for how to get a host list by parsing a url, maybe it's useful for someone.

Also I'd like to use threading for the part in lines 581-597 so that runs in parallel instead of in series. This may require changes to the ServerSetup class though.

Code: (python)
        for host in hosts:
           setup = ServerSetup(handler=handler, username=username, host=host,\
               port=ssh_port, buf_size=buf_size, timeout=timeout,\
               pp_port=pp_port, password=password, auto_add=auto_add,\
               pp_secret=pp_secret)
           try:
               workers = setup.run()
           except IOError:
               logger.error("Communication for '%s@%s' failed!", username,
                   host)
               rm.append(host)
           else:
               count += workers
               logger.info("%d ppserver(s) now running on '%s'.", workers,\
                   host)
               if not workers:
                   rm.append(host)
 

The pp_task.py script is meant to be changed for each job that you require. It should be in the same directory (or in PYHONPATH) as pp_setup.py as it is imported by the latter. It's a simple interface to define everything the server needs to submit its tasks. If your callback function requires more arguments they should be put before server, result in the function definition.

I have absolutely not tested this script for Windows at all, if anyone ever needs that I'm happy for any hints/code snippets.

I also have not tested the --job-only or --servers-only options very much. I guess for --servers-only to work the remote ppservers would really have to be started with the -a option and you'd have to specify --hosts "*" for the pp_setup.py script afterwards or so.

Also, please be patient for the jobs that you specify to be worked on. If you have many jobs it takes a while to submit all of them, plus parallel python seems to need a while to communicate with all remote ppservers. So just wait when nothing happens.

Enjoy, and comment Smiley

* client_task.py (0.28 KB - downloaded 98 times.)
* pp_setup.py (22.89 KB - downloaded 93 times.)
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.16 | SMF © 2011, Simple Machines Valid XHTML 1.0! Valid CSS!
Nutrition facts and analysis