 Show Posts
|
|
Pages: [1] 2
|
|
1
|
Python Forums / Parallel Python Forum / Minor bug in ppcommon.py
|
on: September 06, 2011, 02:41:53 AM
|
|
In v 1.6.1 ppcommon.py line 38-43
Originally:
def start_thread(name, target, args=(), kwargs={}, daemon=True):
"""Starts a thread"""
thread = threading.Thread(name=name, target=target, args=args, kwargs=kwargs)
thread.daemon = True
thread.start()
return thread
This ignores the daemon parameter passed to the function and always sets it to true.
The more correct form would be
def start_thread(name, target, args=(), kwargs={}, daemon=True):
"""Starts a thread"""
thread = threading.Thread(name=name, target=target, args=args, kwargs=kwargs)
thread.daemon = daemon
thread.start()
I dont think this actually makes any difference to anything, but I thought I would report it for completeness!
return thread
|
|
|
|
|
3
|
Python Forums / Parallel Python Forum / subprocess
|
on: August 22, 2011, 04:28:33 AM
|
I am trying to use the subprocess module in the standard library without luck For example: import pp def myfunc(): subprocess.call("hostname") if __name__ == "__main__": job_server = pp.Server() j = job_server.submit(myfunc, modules=("subprocess",)) j() This gives the error: Exception in thread run_local: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 477, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.6/dist-packages/pp.py", line 721, in _run_local job.finalize(sresult) UnboundLocalError: local variable 'sresult' referenced before assignment I know that the error in PP will be fixed in the next version ( http://www.parallelpython.com/component/option,com_smf/Itemid,29/topic,604.0), but it also seems like PP clients cant spawn new processes. I guess this may also affect multiprocessing and threading modules as well. |
|
|
|
|
4
|
Python Forums / Parallel Python Extensions and Applications / pyumpf
|
on: August 19, 2011, 07:37:54 AM
|
I've made my tool available at https://github.com/afaulconbridge/pyumpfpyumpf ====== Python Unified Multiprocessing Parallel Functions This is a small library that aims to bring together the standard library map/reduce functions with both the multiprocessing module in the standard library and the PP module ( www.parallelpython.com). Each has their own niche, and there own advantages and disadvantages. The map/reduce functions in the standard library are good for implementing a functional paradigm, but are limited by the GIL to only run one task at a time. The multiprocessing module is good for single machines with multiple CPUs, but can be awkward to debug and use. The PP module is great for clusters, but is even more awkward to debug and use. Plus all of these have different APIs, and so cannot be drop-in replacements. pyumpf gets around this by providing a unified interface via umpf.map and umpf.reduce that can easily be expanded to use multiprocessing or PP when needed, but also collapsed to the built-in map/reduce for simpler debugging. To install: python setup.py install For an example, see umpf_test.py Note that this example gives PP a poor performance. This is because it is running on a single machine, and this example is too small to make best use of the parallelism. Ideally, tasks should run for several seconds each and have easily pickleable arguments and returned values. |
|
|
|
|
6
|
Python Forums / Parallel Python Forum / Re: pp not using all workers
|
on: August 19, 2011, 07:18:22 AM
|
|
I've noticed that sometimes jobs are so short that an earlier worker becomes available again before farming jobs to all workers is completed. Try making your test jobs more intensive, or adding sleep commands.
|
|
|
|
|
7
|
Python Forums / Parallel Python Forum / Re: Open ended job queue.
|
on: August 19, 2011, 07:15:58 AM
|
|
Yes. You can append new jobs in your loop. E.g
jobs = [(m, job_server.submit(run_dydc,(m,), (), ("subprocess",))) for m in members]
for m, job in jobs:
job()
print "simulation # ", m, "is done"
if need_to_do_more_work:
more_work = get_more_work()
for new_m in more_work:
jobs.append((new_m, job_server.submit(run_dydc,(new_m,), (), ("subprocess",))
This is totally untested, but...
|
|
|
|
|
8
|
Python Forums / Parallel Python Forum / Re: ppmap problems
|
on: December 01, 2009, 04:45:12 AM
|
Ah ha! I have a solution which seems to work; using the inspect module to look through the stack and add all globals. This is mypp.py import pp
import inspect
def ppmap(func, args, ppservers=('*',), ncpus='autodetect', secret=pp.Server.default_secret, stats=False):
""""
This is a wrapper for the Parallel Python processing.
ppservers is a tuple of servers to use
func is function
args is an iterable of lists of arguments to func
secret is the secret value set on the clients
stats is a boolean whether to print
retuns an iterator over the results
"""
job_server = pp.Server(ncpus=ncpus, ppservers=tuple(ppservers), secret=secret)
jobs = []
#add all the things from all stacks into one uber-global namespace
#probably a bad idea, but meh
stack = reversed(inspect.stack())
globalthings = {}
for thing in stack:
for stuff in thing[0].f_globals:
globalthings[stuff] = thing[0].f_globals[stuff]
#some cleaning incase the garbage collector has trouble
del thing
del stack
#globalthings = globals()
for arg in args:
try:
arg = tuple(arg)
except TypeError:
arg = (arg,)
#need to manually specify pp module because pp.py ignores it in global space
jobs.append(job_server.submit(func, arg, modules=("pp",), globals=globalthings))
# do this separately to make sure they are all sent before getting results...
for job in jobs:
yield job()
if stats:
job_server.print_stats()
#some cleaning incase the garbage collector has trouble
del globalthings
if __name__=="__main__":
def myboris(n):
return 2**n
def testfunction(n):
return myboris(n)
for thing in ppmap(testfunction, xrange(10)):
print "-"*50
print thing
print "-"*50
And this is an example of it being used in another module: import mypp
import math
def testfunction(n):
return math.factorial(n)
if __name__ == "__main__":
for thing in mypp.ppmap(testfunction, xrange(10)):
print "-"*50
print thing
print "-"*50
I'm sure this is not particularly efficient, but it seems to work in my limited tests. |
|
|
|
|
9
|
Python Forums / Parallel Python Forum / Re: ppmap problems
|
on: December 01, 2009, 03:54:24 AM
|
|
I could be more specific with the globals variable, but I'm trying to be generic so that the dependent functions and modules don't have to be hand-calculated and manually passed in.
It turns out this problem was that pp.py does not import pp if it is in the global space. The solution was to pass pp to each job as a module to import.
This however, the leads to another problem. globals() isnt global, it returns the namespace of the module it is called in. So in my case, the globals() inside my ppmap function returns the namespace of my mypp module, which is not useful.
I could either demand that anything that uses it passes globals() in, or try to find a way to get the global globals() which would be neater.
|
|
|
|
|
10
|
Python Forums / Parallel Python Forum / Re: Passing optional args with PP
|
on: November 30, 2009, 07:42:03 AM
|
Could you use the "globals" parameter, e.g.: job_server.submit(func, arg, globals={'write_to_disk_default':True}) Then inside the function, define the defaults as globals with the actual function default values be a value to check the global if not specified, e.g. write_to_disk_default = False
def somefunc(write_to_disk=None):
if write_to_disk is None:
write_to_disk=write_to_disk_default |
|
|
|
|
11
|
Python Forums / Parallel Python Forum / ppmap problems
|
on: November 30, 2009, 07:34:16 AM
|
I am trying to implement a drop-in replacement for the built-in map command. Or at least, as drop-in as possible. To this end, I have created a new script mypp.py import pp
def ppmap(func, args, ppservers=('*',), ncpus='autodetect', secret=pp.Server.default_secret, stats=False):
""""
This is a wrapper for the Parallel Python processing.
ppservers is a tuple of servers to use
func is function
args is an iterable of lists of arguments to func
secret is the secret value set on the clients
stats is a boolean whether to print
retuns an iterator over the results
"""
job_server = pp.Server(ncpus=ncpus, ppservers=tuple(ppservers), secret=secret)
jobs = []
for arg in args:
try:
arg = tuple(arg)
except TypeError:
arg = (arg,)
jobs.append(job_server.submit(func, arg, globals=globals()))
# do this separately to make sure they are all sent before getting results...
for job in jobs:
yield job()
if stats:
job_server.print_stats()
if __name__=="__main__":
def testfunction(n):
return 2**n
for thing in ppmap(testfunction, xrange(10)):
print "-"*50
print thing
print "-"*50
I am trying to use the globals argument to avoid having to manually pass require functions & modules. However, when I use the globals=globals() parameter, I get an import error for every job: An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
1
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
2
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
4
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
8
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
16
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
32
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
64
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
128
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
256
--------------------------------------------------
An error has occured during the function import
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/ppworker.py", line 86, in run
eval(__fobj)
File "<string>", line 1, in <module>
NameError: name 'pp' is not defined
--------------------------------------------------
512
--------------------------------------------------
Oddly, I still seem to get the right answers back along with the NameErrors. Any idea what is causing this, and how I can fix it? |
|
|
|
|
12
|
Python Forums / Parallel Python Forum / Re: thread.error: can't start new thread
|
on: November 30, 2009, 07:20:48 AM
|
|
As I suspected, this wasnt a PP problem.
For future reference in case anyone is interested, Sun Grid Engine assigns a stack size equal to the assigned memory if using a hard limit. This means when a new thread is created, there is not enough memory for it. Adding the following fixed it for me: -l h_stack=8M
|
|
|
|
|
13
|
Python Forums / Parallel Python Forum / thread.error: can't start new thread
|
on: November 25, 2009, 10:23:28 AM
|
|
This may or may not be a PP problem itself, but its surfacing while using PP so I thought I'd see if anyone has any answers.
I am using a small cluster running Sun Grid Engine. I submit jobs which start several ppserver.py instances on several machines (depending on load etc) and a job which starts the main python script to coordinate them. If I do this by hand, then it works fine. If I do it via Sun Grid Engine, then I get the following for a ppserver.py
2009-11-25 18:02:44,747 DEBUG Creating server instance (pp-1.5.7)
2009-11-25 18:02:44,872 INFO pp local server started with 1 workers
2009-11-25 18:02:44,872 DEBUG Strarting network server interface=0.0.0.0 port=60000
2009-11-25 18:02:44,872 DEBUG ppserver will exit in 120 seconds if no connections with clients exist
Traceback (most recent call last):
File "/scratch/pp-1.5.7/ppserver.py", line 329, in <module>
server = _NetworkServer(**args)
File "/scratch/pp-1.5.7/ppserver.py", line 87, in __init__
thread.start_new_thread(self.check_timeout, ())
thread.error: can't start new thread
This is from a command: python2.6 /scratch/pp-1.5.7/ppserver.py -a -d -t 120 -n 2 -w 1 -r
And from the example python2.6 /scratch/pp-1.5.7/examples/sum_primes_functor.py 1
I get:
Usage: python sum_primes.py [ncpus]
[ncpus] - the number of workers to run in parallel,
if omitted it will be set to the number of processors in the system
Traceback (most recent call last):
File "/scratch/pp-1.5.7/examples/sum_primes_functor.py", line 47, in <module>
job_server = pp.Server(ncpus, ppservers=ppservers)
File "/scratch/pp-1.5.7/pp.py", line 370, in __init__
self.__connect()
File "/scratch/pp-1.5.7/pp.py", line 599, in __connect
thread.start_new_thread(discover.run, ppserver)
thread.error: can't start new thread
Again, if I run it manually it works, so it's probably sun grid engine rather than parallel python.
It seems to be unable to start new threads, but running by hand shows that it does work so cant be an OS/hardware thing. It also shows its probably not PP, but if anyone has any ideas, Id like to hear them
Thanks!
|
|
|
|
|
Forums 
