Hi,

I have seen this issue on a windows vista running python 2.6.1 and pp 1.5.7.
I have seen this when calling the destroy() method on a ppserver.

Krb

Is is prohibited to use print >> sys.__stdout__ from the funtions that pp's workers run.

for more info on the error related to using such a statement, please see:

http://www.parallelpython.com/component/option,com_smf/Itemid,29/topic,385.0

Ok, so found out why my app is getting a long ranther than an int. And I have been able to reproduce the same problem using pp's supplied examples.

if the function that is run by the pp's submit method prints directly to sys.__stdout__ then you can see the above mentioned behaviour.

so for example, in sum_primes_functor.py example supplied with pp, if yo add 'print >> sys.__stdout__, n , "is prime";' at line 26 (just before return True), then you'll see the same error as reported by my app.

So is it prohibited to use statements like 'print >> sys.__stdout__' from pp's worker.

Now instead of 'print >> sys.__stdout__' just print >> sys.stdout or just plain print is used, then nothing happens, ie everything works as usual.

Found a bug (I think) in pptransport.PipeTransport.receive line 130.

Specifically struct's "!Q" format string used in pack and unpack, represents "unsigned long long for c and long for python".

So msg_len can contain a value larger than 32 bit int and r.read expects a 32 bit int. And this is causing a problem.

Although I have to say, I am not sending more than 2^32 bytes of data. Not sure where it is comming from??


   

Hi,
I get an error when I start a job in pp. I am sure something is wrong on the application side (rather than pp side), as this error is not generated in all use cases. I would like to know why pp would generate such a message.

Thanks,
krb

Traceback (most recent call last):
  File "g:\Tools\localWin\python26\lib\site-packages\pp.py", line 762, in __run
    sresult = worker.t.receive()
  File "g:\Tools\localWin\python26\lib\site-packages\pptransport.py", line 133,
in receive
    msg = self.r.read(msg_len)
OverflowError: long int too large to convert to int
Unhandled exception in thread started by
<bound method Server.__run of <pp.Server object at 0x01FAED50>>
Traceback (most recent call last):
  File "g:\Tools\localWin\python26\lib\site-packages\pp.py", line 762, in __run
    sresult = worker.t.receive()
  File "g:\Tools\localWin\python26\lib\site-packages\pptransport.py", line 133,
in receive
    msg = self.r.read(msg_len)
OverflowError: long int too large to convert to int
Unhandled exception in thread started by <bound method Server.__run of <pp.Serve
r object at 0x01FAED50>>

>>That is why I chose to read the gdal objects in chunks and dump them to files. So each chunk is an array of smaller size (for instance: >>23,20,20)

So once you dumped your gdal object to file, you don't consume lot of memory, right?
I think, instead of dumping it to a file you can send it to pp_server. (try cStringIO's file like object to print everything and then pass its value).

And since you submit a job sequentially, you cannot be consuming whole lot of memory.

I did something similar for my current project, only difference being my objects are not 200Mb, but I have lot of them about 3000-20000.

ok I'll break it down into steps and see if my understanding of your program is same as yours.

• 1 you read in a gdal object
• 2 convert the gdal data, which is on disk, to numpy array which is in memory
• 3 write the numpy data to a pickle file
• 4 pass the pickle file to pp_server

ok, so if the above is correct then what I am saying is

• 1 you read in a gdal object
• 2 convert the gdal data, which is on disk, to numpy array which is in memory
• 3 pass the numpy array to pp_server
• 4 delete the numpy array, so keep your memory usage to a minimum, just after you submit a job

so you submit a job and delete the numpy array. When pp is done sending the numpy array it will no longer refer to the numpy array so python will release the memory associated with it.

As you submit jobs sequentially, you will be dealing with 1 chunk of numpy array at a time (pp may still be referencing it, and it depends on how wide your pp server is).

why dump the pickle to a file and pass it.
pp does that, I mean (I think) it uses 'dumps' and then passes that data around to the servers (local or on grid).

so if you happen to convert your gdal data into numpy array, then no need to use the file based approach.
I am, of course, assuming that the gdal to numpy conversion consumes most of the memory. So, if you write it to a file or send it to a pp server does it matter??

just curious on what your gdal object is?
is it a file pointer(file handle) type??

I had similar problems, in my case I was trying to pass an unpickleable object via pp's submit routine.
Looks like your PySwig object cannot be pickled, so yeah use a pickle-able object.
I know that numpy array's can be pickled, so use them instead and try.

I have noticed that if I use the -d switch after all the other options, then ppserver is not using it.

say 'ppserver.py -p 951357 -w 2  -s somesecret -d' doesn't put ppserver in debug mode.

is this expected?

Hi,

I have been trying to figure out how the -i switch works.

Say, I use -i 192.168.0.0 will pp only accept connections from a machine with  ip address 192.168.0.0 or any machine with ip address greater than 192.168.0.0 or anything else?

Thanks

pp's examples are great to demonstrate its power, but most of the users would use pp where multiple modules/classes are involved. It would be great to have an example which involves multiple modules/classes using pp.

if there is an example like this, please point me to it.

try passing sanitize as depended function when you submit a job, look at the api for more info

Ok so I figured out why pp is importing the module from which it is used.
After instantiating a pp server I am importing a module (user supplied one) from a function.

I though this might be causing PP to think that the calling module changed and hence importing it.

So I moved this function into another module and noticed that this module in which pp is instantiated in not imported anymore.