pp and py2exe

(1/1)

wollmich:
Hello!

I use Python 2.5.1 and Numpy 1.0.4 on Windows XP Prof. SP2.

I've a problem using Parallel Python with py2exe

Simple example:

filename: test_pp.py
Code: (python)
import pp
import numpy
 
 
print 'vor Server'
job_server = pp.Server()
print 'nach Server'
 
 
def test2(i):
   a = numpy.linalg.inv(numpy.identity(1000))
   print 'hallo %d' %i
 
 
jobs =[]
for i in xrange(100):
   job = job_server.submit(test2,(i,),(),('numpy',))
   jobs.append(job)
print 'gestartet'
 
for job in jobs:
   job()
print 'ende'
 

py2exe file:
Code: (python)
from distutils.core import setup
import py2exe
 
setup(console=['test_pp.py'])
 

Please help me?

Thanks and greatings from switzerland
wollmich

Alkemir:
Perharps it would be easier if you posted the information on
what's going wrong?

Vitalii:
I've tried py2exe with sum_primes.py and in fact it does not work:

Code: (bash)
L:\PP\examples\dist>sum_primes.exe
Usage: python sum_primes.py [ncpus]
   [ncpus] - the number of workers to run in parallel,
   if omitted it will be set to the number of processors in the system
 
Traceback (most recent call last):
 File "sum_primes.py", line 49, in <module>
 File "pp.pyc", line 235, in __init__
 File "pp.pyc", line 362, in set_ncpus
 File "pp.pyc", line 117, in __init__
 File "pptransport.pyc", line 115, in receive
ValueError: invalid literal for int() with base 10: 'Usage: python sum_primes.py
[ncpus]\n'

From the error message I see why it doesn't work: instead of running ppworker.py it tries to execute itself again (perhaps because py2exe rewrote the paths and bundled ppworker.py inside).
At this point there is no obvious solution to this problem.

Regards,
Vitalii

klokan:
Hi Vitalii and Wollmich,

I managed to pack the sum_primes.py example and parallel python into the exe with py2exe. It is problematic but possible. I made a patch for that: an example and documentation is here: http://github.com/klokan/py2exepp

Better solution would be to add a function similar to multiprocessing.freeze_support() into pp.
Programmers would have to add this one into the main() function of their program, if they would like to support freeze via py2exe or other alternatives. Such function would scan the argv and start ppworker instead of the main program in case it detects the relevant arguments.

Unfortunately I am getting EOFError when I try that, because it seems that the 'unbuffered':True in py2exe is not running properly. Once py2exe supports correctly 'unbuffered' mode it should run well, till then the solution in github is a (dirty) workaround.

Greetings from Switzerland as well ;-).

Klokan Petr Pridal

klokan:
Hi,

I have reported the problem to py2exe mailing-list, so more info should appear here:
http://tinyurl.com/yarxgll

Klokan

Navigation

[0] Message Index