Parallel Python

OliverKing

Jr. Python

Karma: 0
Posts: 5

Job not returning data: crashes with OverflowError

« on: August 05, 2009, 04:43:58 PM »

Hi,

I have a problem that is driving me up the wall. I'm using pp version 1.5.7, with Python 2.5.4, on Mac OS X 10.5.7. The code basically farms out a fairly repetitive task to the 2 processors on my macbook. This task calls a self-written C-library, using ctypes 1.0.3, to do some intensive processing. Here is a skeleton of the code:

Code:

def repetitive_task(parameters for task):
# Create C library object
libcc = ctypes.CDLL("calclib.so")
# Allocate some arrays using numpy - these are filled by the C library
array1 = numpy.zeros((N,),dtype=int)
array2 = numpy.zeros((N,),dtype=float)
libcc.processor_intensive_task(array1.ctypes.data, array2.ctypes.data, other parameters)
return (array1, array2, other parameters)

def call_repetitive_tasks(parameters):
ppservers = ()
job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"
jobs = []
for m in range(number_of_tasks):
jobs.append(job_server.submit(repetitive_task,(parameters),(),("numpy","ctypes")))
for job in jobs:
result = job()
array1 = result[0]
array2 = result[1]
#retrieve other parameters as well

I call the

Code:

call_repetitive_tasks()

function from another python task. By writing to log files I know that the C library executes correctly, and that the

Code:

repetitive_task()

function also executes correctly with its arrays correctly filled with data.

However, the process then dies - the

Code:

result = job()

line does not execute successfully. The error message I get when it dies is:

Code:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pp.py", line 762, in __run
sresult = worker.t.receive()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pptransport.py", line 133, in receive
msg = self.r.read(msg_len)
OverflowError: long int too large to convert to int
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0x189e4b0>>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pp.py", line 771, in __run
job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

It returns this error message twice - once for each processor in my macbook - and then hangs.

Does anyone know what is causing this behaviour?

Thanks,
Oliver


	Logged

OliverKing

Jr. Python

Karma: 0
Posts: 5

Re: Job not returning data: crashes with OverflowError

« Reply #1 on: August 11, 2009, 04:01:50 PM »

It seems that the crash had something to do with using numpy data types. It now works, but takes a long time to return data.


	Logged

OliverKing

Jr. Python

Karma: 0
Posts: 5

Re: Job not returning data: crashes with OverflowError

« Reply #2 on: August 11, 2009, 05:19:22 PM »

As I mentioned in my previous post, I can now call pp without the overflow error. However, there is a large delay between the completion of a job and the retrieval of the the result.

I modified the code to record job submit, start, stop, and retrieve times. The results are (omitting some of the less significant digits):

Code:

Submitting job: 1
Submit time: 0.0001
Submitting job: 2
Submit time: 0.01
Submitting job: 3
Submit time: 0.0149
Submitting job: 4
Submit time: 0.0153
Processing job: 2
Start time: 0.56
End time: 5.01
Processing job 1
Start time: 0.56
End time: 5.37
Retrieving job: 1
Retrieve time: 63.896
Retrieving job: 2
Retrieve time: 64.096
Processing job: 3
Start time: 63.65
End time: 68.60
Processing job: 4
Start time: 63.72
End time: 68.94
Retrieving job: 3
Retrieve time: 130.367
Retrieving job: 4
Retrieve time: 130.794

As you can see, there is a delay of nearly 60s between the completion of the first two jobs and their data being read out. The command which writes the log entry is the last line before the return command.

What could be causing this delay? It seems to occupy almost all of my CPU cycles for the duration of the delay, according to the Mac Activity Monitor.


	Logged

OliverKing

Jr. Python

Karma: 0
Posts: 5

Re: Job not returning data: crashes with OverflowError

« Reply #3 on: August 12, 2009, 10:47:57 AM »

I've discovered that the delay is due to the size of the numpy arrays being returned by the jobs. Reducing the array sizes drops the time by a huge factor.

I did not find this problem when I used python lists of the same size, rather than numpy arrays. Does pickle have trouble with numpy arrays?


	Logged

OliverKing

Jr. Python

Karma: 0
Posts: 5

Re: Job not returning data: crashes with OverflowError

« Reply #4 on: August 12, 2009, 02:00:56 PM »

In the end, the fastest solution was to use ctypes abstract data types instead of numpy arrays.

i.e. instead of

Code:

array1 = numpy.zeros((N,),dtype=int)
array2 = numpy.zeros((N,),dtype=float)
libcc.processor_intensive_task(array1.ctypes.data, array2.ctypes.data, other parameters)
return (array1, array2, other parameters)

use

Code:

type1 = ctypes.c_long * N
array1 = type1()
type2 = ctypes.c_double * N
array2 = type2()
libcc.processor_intensive_task(ctypes.byref(array1), ctypes.byref(array2), other parameters)
return (array1[:], array2[:], other parameters)

I'm sure it would be much faster to return array1 and array2, but pickle can't handle ctypes abstract data types and my Python foo isn't yet powerful enough for me to figure out how to write a __reduce__() function.


	Logged

Pages: [1]

« previous next »

	Author	Topic: Job not returning data: crashes with OverflowError (Read 3520 times)
0 Members and 1 Guest are viewing this topic.