 Show Posts
|
|
Pages: [1]
|
|
1
|
Python Forums / Parallel Python Forum / Method of connecting to hosts
|
on: January 03, 2011, 06:47:46 AM
|
|
Hi, I have the following issue:
I retrieve a list of hosts by parsing a website and use that list to start a local pp.Server. Now, on some of those hosts ppserver.py processes will be running and on some there won't. My problem is that the local server seems to only try two of those hosts (which do not run any ppserver.py) and since it fails to connect, retries every 5s. It never loops through the whole list to try and find ppservers on the other hosts.
Can this be improved?
|
|
|
|
|
3
|
Python Forums / Parallel Python Forum / Re: pp 1.5.7 and numpy bug
|
on: January 25, 2010, 01:49:28 AM
|
Hmm, this line makes me suspicious: File "/usr/share/pyshared/numpy/lib/type_check.py", line 8, in <module>
import numpy.core.numeric as _nx
In my personal code using import ... as ... did not work with parallel python, maybe that's the same problem with numpy. Can you edit the numpy file accordingly and try again? |
|
|
|
|
4
|
Python Forums / Parallel Python Forum / Re: pp 1.5.7 crash
|
on: January 25, 2010, 01:41:47 AM
|
I got this kind of error from using a simple subprocess call, if you use subprocess you should invoke it the following way: p = subprocess.Popen(["prog", "arg1"], stdin=subprocess.PIPE, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
and then either call wait() on p or communicate(). |
|
|
|
|
6
|
Python Forums / Parallel Python Forum / Re: subprocess or pp error?
|
on: January 12, 2010, 09:05:55 AM
|
I tried around some more and I have the following suggestions for you, first of all I had to not use shell=True (default is False anyway) when trying this and I would insert p.wait() just to make sure the command ends before proceeding and so that you get p.returncode even if you're not interested in the stdout. p = subprocess.Popen(["./prog", ""], stdin=subprocess.PIPE, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p.wait()
Hope that helps. |
|
|
|
|
7
|
Python Forums / Parallel Python Forum / Re: subprocess or pp error?
|
on: January 11, 2010, 02:43:32 AM
|
You are absolutely right! If one doesn't resort to the subprocess convenience functions but instead connects to the command's stdin, stdout, and stderr it works. E.g.: p = subprocess.Popen(["ls", "-l"], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
(stdout, stderr) = p.communicate()
print p.returncode
print stdout
|
|
|
|
|
9
|
Python Forums / Parallel Python Extensions and Applications / Remote ppserver setup script
|
on: January 09, 2010, 05:22:49 PM
|
Hi all, I've attached two scripts that can be used to set up a number of ppserver(s) on remote hosts which will then be addressed by a local server with 0 workers. pp_setup.py logs into the remote hosts using ssh via the paramiko module, checks the cpus online there as well as the current load and then sets up a ppserver with a number of workers accordingly. You should at least start the pp_setup.py script interactively so that you can type in a password in the shell if needed. You can however specify a logfile and simply send it to the background after entering the password and then disown it if you need to run this on a remote machine as well. Also note that in line 549 the call to CLAMVParser is something rather internal. I changed the CLAMVParser because I did not want to reveal how to use our university's resources  but it's a skeleton now for how to get a host list by parsing a url, maybe it's useful for someone. Also I'd like to use threading for the part in lines 581-597 so that runs in parallel instead of in series. This may require changes to the ServerSetup class though. for host in hosts:
setup = ServerSetup(handler=handler, username=username, host=host,\
port=ssh_port, buf_size=buf_size, timeout=timeout,\
pp_port=pp_port, password=password, auto_add=auto_add,\
pp_secret=pp_secret)
try:
workers = setup.run()
except IOError:
logger.error("Communication for '%s@%s' failed!", username,
host)
rm.append(host)
else:
count += workers
logger.info("%d ppserver(s) now running on '%s'.", workers,\
host)
if not workers:
rm.append(host)
The pp_task.py script is meant to be changed for each job that you require. It should be in the same directory (or in PYHONPATH) as pp_setup.py as it is imported by the latter. It's a simple interface to define everything the server needs to submit its tasks. If your callback function requires more arguments they should be put before server, result in the function definition. I have absolutely not tested this script for Windows at all, if anyone ever needs that I'm happy for any hints/code snippets. I also have not tested the --job-only or --servers-only options very much. I guess for --servers-only to work the remote ppservers would really have to be started with the -a option and you'd have to specify --hosts "*" for the pp_setup.py script afterwards or so. Also, please be patient for the jobs that you specify to be worked on. If you have many jobs it takes a while to submit all of them, plus parallel python seems to need a while to communicate with all remote ppservers. So just wait when nothing happens. Enjoy, and comment  |
|
|
|
|
10
|
Python Forums / Parallel Python Forum / Re: job server reports done, when silently nothing happened.
|
on: January 08, 2010, 04:10:30 AM
|
Actually, it doesn't work smoothly, I only thought it did because the jobs I wanted to be performed were started nicely but in reality there is some kind of communication problem between the remote server and its workers. And the error only occurs when I do a subprocess call. The following is my test function: def perform(argv):
"""
main function to perform tasks
"""
time.sleep(10)
subprocess.check_call(["ls"])
argv[0]**argv[0]
return 666
Without the subprocess call everything works flawlessly but with the subprocess call the below error occurs. Strangely, the error is thrown before the subprocess call actually returns (tested with something that takes longer than "ls"). So maybe when a system call is used in a worker thread it returns immediately? I'm confused and don't have much experience with threads. Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/pp.py", line 762, in __run
sresult = worker.t.receive()
File "/usr/lib/python2.6/site-packages/pptransport.py", line 133, in receive
msg = self.r.read(msg_len)
OverflowError: long int too large to convert to int
Unhandled exception in thread started by <bound method _NetworkServer.__run of <__main__._NetworkServer object at 0xb754c66c>>
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/pp.py", line 771, in __run
job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment
I wanted to use parallel python to just start a lot of programs I had compiled in C++. I'm going to experiment a bit with other forms of system calls. |
|
|
|
|
12
|
Python Forums / Parallel Python Forum / job server reports done, when silently nothing happened.
|
on: December 16, 2009, 07:28:21 AM
|
Hello, I suppose my problem is the following: I use parallel python not to compute something within python but my jobs look like this: def perform(filename, rob_type, noise):
"""
main function to perform tasks
"""
if noise:
cmd = ["f_simulation", "-i", "8", "-m", "20", "-o", "8",
"-t", "0.007", "-s", "0.0001", "-e", "100000", "-d",
"-f", filename, rob_type, noise]
else:
cmd = ["f_simulation", "-i", "8", "-m", "20", "-o", "8",
"-t", "0.007", "-s", "0.0001", "-e", "100000", "-d",
"-f", filename, rob_type]
subprocess.check_call(cmd)
print "Finished", " ".join(cmd)
return None
When I call this function normally from python I have no problems but when I run it over the cluster the job server reports all jobs being submit and everything done within seconds. But the subprocesses are not actually executed. I should add that I submit this function as an import from another python module, i.e., with the following code I set up my job server: pp_servers = list()
for host in hosts:
pp_servers.append("%s:%d" % (host, pp_port))
pp_servers = tuple(pp_servers)
job_server = pp.Server(ncpus=0, ppservers=pp_servers, secret=pp_secret,
proto=pickle.HIGHEST_PROTOCOL, loglevel=log_level,
logstream=log_stream)
logger.debug("creating template")
cb_args = [job_server]
cb_args.extend([arg for arg in pp_tasks.callbackargs])
cb_args = tuple(cb_args)
fn = pp.Template(job_server=job_server, func=pp_tasks.perform,
depfuncs=pp_tasks.depfuncs, modules=pp_tasks.modules,
callback=pp_tasks.callback, callbackargs=cb_args,
group=pp_tasks.group, globals=pp_tasks.glob)
logger.debug("making arguments")
arguments = pp_tasks.make_args()
jobs = list()
for args in arguments:
jobs.append(fn.submit(args))
job_server.wait()
logger.info("results ready")
job_server.print_stats()
Should I not import the task to be done directly but use a proxy within the same script as has been suggested for classes elsewhere? Suggestions are welcome |
|
|
|
|
13
|
Python Forums / Parallel Python Forum / Re: Two Feature Requests
|
on: December 12, 2009, 05:42:07 PM
|
|
To change the logging of ppserver to report back to the job server or at least log to files, were features that I immediately wanted to have as well. Since you said you were considering this, are you working on that already? At the moment I'm considering three ways to go ahead for myself: Sub-class to add this capability, build similar capabilities using multiprocessing (as I'm fortunate to have Python 2.6 on the cluster I want to use but then it would be considerably more work), or if you accept that at all, I could add in the capabilities and you can merge them as you see fit.
|
|
|
|
|
Forum 
