Parallel Python

krb

Full Python

Karma: 0
Posts: 23

OverflowError: long int too large to convert to int

« on: October 06, 2009, 05:37:17 AM »

Hi,
I get an error when I start a job in pp. I am sure something is wrong on the application side (rather than pp side), as this error is not generated in all use cases. I would like to know why pp would generate such a message.

Thanks,
krb

Traceback (most recent call last):
File "g:\Tools\localWin\python26\lib\site-packages\pp.py", line 762, in __run
sresult = worker.t.receive()
File "g:\Tools\localWin\python26\lib\site-packages\pptransport.py", line 133,
in receive
msg = self.r.read(msg_len)
OverflowError: long int too large to convert to int
Unhandled exception in thread started by
<bound method Server.__run of <pp.Server object at 0x01FAED50>>
Traceback (most recent call last):
File "g:\Tools\localWin\python26\lib\site-packages\pp.py", line 762, in __run
sresult = worker.t.receive()
File "g:\Tools\localWin\python26\lib\site-packages\pptransport.py", line 133,
in receive
msg = self.r.read(msg_len)
OverflowError: long int too large to convert to int
Unhandled exception in thread started by <bound method Server.__run of <pp.Serve
r object at 0x01FAED50>>


	Logged

krb

Full Python

Karma: 0
Posts: 23

Re: OverflowError: long int too large to convert to int

« Reply #1 on: October 06, 2009, 06:59:36 AM »

Found a bug (I think) in pptransport.PipeTransport.receive line 130.

Specifically struct's "!Q" format string used in pack and unpack, represents "unsigned long long for c and long for python".

So msg_len can contain a value larger than 32 bit int and r.read expects a 32 bit int. And this is causing a problem.

Although I have to say, I am not sending more than 2^32 bytes of data. Not sure where it is comming from??

Code:

def receive(self, preprocess=None):
size_packed = self.r.read(struct.calcsize("!Q"))
msg_len = struct.unpack("!Q", size_packed)[0]
print type(msg_len), msg_len, size_packed, ; ## krb
if hasattr(self.r, 'fileno'): ## krb
print 'fileno:', self.r.fileno(), ; ## krb
print; ## krb
##print size_packed;
msg = self.r.read(msg_len)
return map(preprocess, (msg, ))[0]


	Logged

krb

Full Python

Karma: 0
Posts: 23

Re: OverflowError: long int too large to convert to int

« Reply #2 on: October 06, 2009, 09:27:10 AM »

Ok, so found out why my app is getting a long ranther than an int. And I have been able to reproduce the same problem using pp's supplied examples.

if the function that is run by the pp's submit method prints directly to sys.__stdout__ then you can see the above mentioned behaviour.

so for example, in sum_primes_functor.py example supplied with pp, if yo add 'print >> sys.__stdout__, n , "is prime";' at line 26 (just before return True), then you'll see the same error as reported by my app.

So is it prohibited to use statements like 'print >> sys.__stdout__' from pp's worker.

Now instead of 'print >> sys.__stdout__' just print >> sys.stdout or just plain print is used, then nothing happens, ie everything works as usual.


« Last Edit: October 06, 2009, 09:30:12 AM by krb »	Logged

Pages: [1]

« previous next »

	Author	Topic: OverflowError: long int too large to convert to int (Read 5378 times)
0 Members and 1 Guest are viewing this topic.