Parallel Python

aditya

Jr. Python

Karma: 0
Posts: 5

Cannot import a module that prints

« on: January 26, 2010, 11:06:19 PM »

I am trying to use parallel python on a quad-core machine. All pp examples run fine on python2.6.

I need to import a third-party module 'moose' for neuronal network modelling for each of the jobs I submit. This module works perfectly fine if I import it in my usual programs i.e. not using parallel python.

For the sake of completeness I mention that it gives the following output on import:
Python 2.6.4 (r264:75706, Jan 16 2010, 17:08:26)
[GCC 4.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import moose
Warning: Failed to find slot: fluxSrc
Warning: Failed to find slot: xGate.lookup
Warning: Failed to find slot: yGate.lookup
Warning: Failed to find slot: zGate.lookup
.:/export/home/aditya
SIMPATH = .:/export/home/aditya
1988 FuncVecs built for the first time
>>>
--------------------------------------------------------------

But I have trouble importing 'moose' when using pp. When I import it via submit() in parallel python I get the following behaviour:

import pp
job_server = pp.Server(ppservers=ppservers)

jobserver gets initialized with ncpus = 4

Now, if I try to import 'moose' while submitting a job:
job_server.submit(runJob, (inputA, averagingi), (analyzeData, ), ('sys', 'moose') )
here 'moose' for example (a neuronal network modelling tool); the submission happens fine

but when I try to get the output
(errormsg, (meanGlomVm_per, meanSomaVm_per, firingrate_per, firingrate2_per, events)) = jobsList[averagingi]()
I get four of below errors and the terminal hangs:
Traceback (most recent call last):
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 762, in __run
sresult = worker.t.receive()
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pptransport.py", line 133, in receive
msg = self.r.read(msg_len)
MemoryError
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0xdc06a50>>
Traceback (most recent call last):
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 771, in __run
job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

-------------------------------------------------------------------

Rather than pass 'moose' to be imported via submit(), I thought it might work if I 'import moose' inside ppworker.py . However if I try that, I get a memory error and the program exits (terminal does not hang).
...
Traceback (most recent call last):
File "respphase_shifting_cluster_fitter2_BBMitralEventsStagSparseSynInput.py", line 23, in <module>
job_server = pp.Server(ppservers=ppservers)
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 336, in __init__
self.set_ncpus(ncpus)
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 494, in set_ncpus
range(ncpus - len(self.__workers))])
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 141, in __init__
self.start()
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 154, in start
self.pid = int(self.t.receive())
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pptransport.py", line 133, in receive
msg = self.r.read(msg_len)
MemoryError

--------------------------------------------------------

I thought that since 'import moose' prints some message to stdout / stderr, this could be the problem.
So I inserted a print 'test' into ppworker.py and sure enough I got a MemoryError as above right on initializing pp.Server(). Then I diverted sys.stdout to a NullStream()
class NullStream:
def write(self, text):
pass
sys.stdout = NullStream()
sys.stderr = NullStream()
print 'test'

After this I no longer got a MemoryError for this print 'test'. However I continue to have the sresult error reported above with importing moose via submit() or if importing moose in ppworker.py .

So just the fact that import moose gives some output is not the cause of the problem.

I really do want to use parallel python due to its simplicity rather than the various mpi python wrappers. But if this can't be sorted out, I will be forced to switch.


	Logged

psederberg

Full Python

Karma: 0
Posts: 14

Re: Cannot import a module that prints

« Reply #1 on: January 27, 2010, 07:42:46 AM »

Hi Aditya:

This sounds frightfully similar to the issues I'm having with code that used to work without a problem:

http://www.parallelpython.com/component/option,com_smf/Itemid,29/topic,413.0

I just downloaded version 1.5.4 of pp and my code works again. Could you try using an older version (I downloaded it with the following):

Code:

cd /tmp
wget http://www.parallelpython.com/downloads/pp/pp-1.5.4.tar.gz
tar xcvf pp-1.5.4.tar.gz

Be sure to update your PYTHONPATH to point at the downloaded and extracted folder (you don't even need to install it b/c it's pure python):

Code:

export PYTHONPATH=/tmp/pp-1.5.4:$PYTHONPATH

Then retry your script.

If it works, this will supply even more evidence that something is funny with 1.5.7...

Best,
Per


	Logged

aditya

Jr. Python

Karma: 0
Posts: 5

Re: Cannot import a module that prints

« Reply #2 on: January 27, 2010, 08:53:06 AM »

Hi Per!
I downloaded pp 1.5.4 and installed it 'python setup.py' install after removing pp* from site-packages. I've got python 2.6 installed.
Unfortunately I get a similar error : but this time the error is more helpful. I note that each string that is routinely printed by a usual 'import moose' is actually being read by pp.py into sresult by worker.t.receive() !

The string in the below case is "Warning: Failed to find slot: fluxSrc"
Why is pp capturing this instead of letting it go to stdout? I get repeats of the same error for each job. After this the terminal hangs!

Literal for int() is: " Warning: Failed to find slot: fluxSrc
"
Traceback (most recent call last):
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 706, in __run
sresult = worker.t.receive()
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pptransport.py", line 116, in receive
msg_len = int(s)
ValueError: invalid literal for int() with base 10: 'Warning: Failed to find slot: fluxSrc\n'
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0x681f310>>
Traceback (most recent call last):
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 715, in __run
job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

-----------------------------------------------------------------

Also now I also get a few deprecation warnings on importing pp.

/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py:34: DeprecationWarning: The popen2 module is deprecated. Use the subprocess module.
import os, thread, popen2, logging, inspect
/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py:35: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sys, sha, types, time, atexit, user
/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pptransport.py:32: DeprecationWarning: the md5 module is deprecated; use hashlib instead
import md5, os, struct, sha, socket, logging
/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py:47: DeprecationWarning: the sets module is deprecated
from sets import Set as set


	Logged

psederberg

Full Python

Karma: 0
Posts: 14

Re: Cannot import a module that prints

« Reply #3 on: January 27, 2010, 08:58:46 AM »

Hi Aditya:

Sorry to send you down a wrong path! I didn't realize you were on Python 2.6, which gave rise to some necessary pp changes put into 1.5.7 to make it compatible (based on the changelog).

As per my subsequent post:

http://www.parallelpython.com/component/option,com_smf/Itemid,29/topic,415.0

perhaps you could try passing in a proto=2 when you instantiate the job server.

This is back using 1.5.7, which allows you to set the pickle protocol.

I hope this one works for you or you'll really be annoyed about having to uninstall and reinstall pp-1.5.7

Best,
Per


	Logged

aditya

Jr. Python

Karma: 0
Posts: 5

Re: Cannot import a module that prints

« Reply #4 on: January 27, 2010, 09:12:38 AM »

Hi Per!

Am happy to be getting inputs from you.
Am back to pp v1.5.7. Tried job_server = pp.Server(proto=2) . Unfortunately same result as before. But what is clear from this exercise (of using v1.5.4) is that a string printed by a module is somehow received by worker.t.receive() instead of going to stdout. Any idea why this should happen?

Traceback (most recent call last):
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 762, in __run
sresult = worker.t.receive()
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pptransport.py", line 133, in receive
msg = self.r.read(msg_len)
MemoryError
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0xbe8cb10>>
Traceback (most recent call last):
File "/home/aditya/Python-2.6.4/lib/python2.6/site-packages/pp.py", line 771, in __run
job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

---------------------------------------------------------------------------

By the way, Per, I replaced moose with numpy in the imports tuple send to submit() and it works fine for me both with proto=2 and without proto=2 in pp.Server(). This could be due to python2.6 for me vs python2.5 for you?


	Logged

psederberg

Full Python

Karma: 0
Posts: 14

Re: Cannot import a module that prints

« Reply #5 on: January 27, 2010, 09:28:59 AM »

Very interesting!

So, it looks like my 1.5.7 bug is python2.5-related. In another thread, someone posted that their errors also went away with the move to python2.6. That's not an option for me at the moment, so I'm glad I can import numpy when I set the protocol to 2.

In your case, I don't have any idea the proper way to handle worker modules that print to stdout, nor to I understand why that would be an issue. I guess we'll have to wait for people with better knowledge of the inner workings of pp to chime in.

Best,
Per


	Logged

Vitalii

Global Moderator
Parallel Python

Karma: 2
Posts: 516

Re: Cannot import a module that prints

« Reply #6 on: January 27, 2010, 11:53:55 PM »

You can print to stdout without doing anything special as long as it is from python code.
For instance you can put print messages inside of you function and that will work (because it PP redirects python stdout).

Code: (python)

import pp
 
def test(i):
    print i
 
server = pp.Server()
 
jobs = []
for i in xrange(5):
    jobs.append(server.submit(test, (i, )))
 
for job in jobs:
    job()

However if something prints to stdout from C/C++ code which is called from python then it won't work since
that code will not follow stdout redirect set by python and will break PP pipes.
It seems like moose module is the case.

The workaround is to place import moose in ppworker.py and surround it with stream redirects:

Code: (python)

savestds = os.dup(1), os.dup(2)
file = open(os.devnull, "w")
os.dup2(file.fileno(), sys.__stdout__.fileno())
os.dup2(file.fileno(), sys.__stderr__.fileno())
 
# place code which writes to C stdout, C stderr here
import moose
 
os.dup2(savestds[0], 1)
os.dup2(savestds[1], 2)
file.close()


	Logged

Join Python Blogs!

Parallel Python

Pages: [1]

« previous next »

	Author	Topic: Cannot import a module that prints (Read 6842 times)
0 Members and 1 Guest are viewing this topic.