Parallel Python Community Forums

I'm trying to use PP in an OO setting.  I could get it to work in the usual (non-OO) way, but PP seems to get confused and can't seem to find a class definition.   Below see my source, followed by the error message - any idea what's wrong??   Thanks!    (I know this is a silly way to implement this; my real project is far more involved, but this code does replicate the errors I'm getting in my larger project.)    I'm fairly new at Python; please forgive me if the error is too obvious..

# *******************************************************************
# File: RafasParallelTotientOO.py
# Author: Rafael Espericueta
# Desc: This program constructs a list of the Euler totient function of
# the first 'Max' positive integers:  [phi(1), phi(2),...,phi(Max)]
# (though in a rather inefficient and silly way!)
# It's computed in 3-number chunks, by submitting parallel jobs.

import sys
sys.path.append('/root/pp-1.6.1')  # This is where my copy of pp lives
import time
import pp

class TotTable:
    def __init__(self,maxnum):
        self.Max = maxnum
        Mx = maxnum+1    # So the array really goes up to and including Max.
       
        # tuple of all parallel python servers to connect with
        ppservers = ()

        # Creates jobserver with automatically detected number of workers
        job_server = pp.Server(ppservers=ppservers)

        print "Starting pp with", job_server.get_ncpus(), "workers"
        start_time = time.time()

        # The following prepares small lists (of length 3) of successive numbers to send
        # to Euler's totient function.  Each 3-item list will be one parallel job.     
        inputs = []
        for n in range(Mx/3):  inputs.append([3*n,3*n+1,3*n+2])
        # Mop up any left-overs
        for n in range(Mx)[3*self.Max/3:]: inputs.append([n])
   
        # Submit all jobs to parallel python
        jobs = [job_server.submit(self.phiList,(input,), (self.phi,self.gcd), ()) for input in inputs]

        job_server.wait()   # wait until all jobs are completed...

        # Concatonate each 3-tuple of completed totients to our totient list
        self.Tot = []
        for job in jobs:   self.Tot += job()

        print "Time elapsed: ", time.time() - start_time, "s"
        job_server.print_stats()
       

    # Outputs the gcd of the two inputs
    def gcd(self,a,b):
        while b != 0:  a, b = b, a%b
        return a

    # Computes Euler's phi function of input value.
    def phi(self,m):
        if m==1:  return 1
        else:
            r = [n for n in range(1,m) if self.gcd(m,n)==1]
            return len(r)

    # Output array is same size of input array.
    def phiList(self,L):
        """Returns 1-dim array of length len(L) containing phi of each entry of input L"""
        return [self.phi(n) for n in L]


# This is the "Main" program :

Euler = TotTable(100)   # the parameter is the maximum number in the totient table

print " *** Now to print out our Table, [phi(1), phi(2),...,phi(Max)] ***"
print Euler.Tot[1:]

# *******************************************************************

Here's the error message:

Starting pp with 8 workers
Traceback (most recent call last):
  File "/root/RafasParallelTotientOO.py", line 68, in <module>
    Euler = TotTable(100)
  File "/root/RafasParallelTotientOO.py", line 36, in __init__
    jobs = [job_server.submit(self.phiList,(input,), (self.phi,self.gcd), ()) for input in inputs]
  File "/root/pp-1.6.1/pp.py", line 458, in submit
    sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
  File "/root/pp-1.6.1/pp.py", line 629, in __dumpsfunc
    sources = [self.__get_source(func) for func in funcs]
  File "/root/pp-1.6.1/pp.py", line 696, in __get_source
    sourcelines = inspect.getsourcelines(func)[0]
  File "/usr/lib/python2.6/inspect.py", line 678, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/lib/python2.6/inspect.py", line 552, in findsource
    raise IOError('could not find class definition')
IOError: could not find class definition

My guess is that you are using an older version of pp ... and that you may have to pass the TotTable to the job_server.
Anyway, I can't tell because the code you posted seems to work for me ... I'm using the new version of pp:
... BTW, I noticed that you probably installed pp manually. Have you considered using easy_install ... files land at the right spot and things seem to work out of the box (at least on my ubuntu linux box).

Output of code follows.
---------------------------------------------
Starting pp with 4 workers
Time elapsed:  0.0219888687134 s
Job execution statistics:
 job count | % of all jobs | job time sum | time per job | job server
        34 |        100.00 |       0.0389 |     0.001144 | local
Time elapsed since server creation 0.0222618579865

 *** Now to print out our Table, [phi(1), phi(2),...,phi(Max)] ***
[1, 1, 2, 2, 4, 2, 6, 4, 6, 4, 10, 4, 12, 6, 8, 8, 16, 6, 18, 8, 12, 10, 22, 8, 20, 12, 18, 12, 28, 8, 30, 16, 20, 16, 24, 12, 36, 18, 24, 16, 40, 12, 42, 20, 24, 22, 46, 16, 42, 20, 32, 24, 52, 18, 40, 24, 36, 28, 58, 16, 60, 30, 36, 32, 48, 20, 66, 32, 44, 24, 70, 24, 72, 36, 40, 36, 60, 24, 78, 32, 54, 40, 82, 24, 64, 42, 56, 40, 88, 24, 72, 44, 60, 46, 72, 32, 96, 42, 40]

Thanks for giving it a test run.   I'm glad to hear it worked for you!

I was using pp version 1.6.1.   I too am running this on an Ubuntu system, using python 2.6.
 
After installing "easy_install" and then reinstalled pp using it, I then commented out the line:   
    # sys.path.append('/root/pp-1.6.1')

However, I get the same error messages as before...   :-(

Any other ideas?

I also can confirm that your code worked for me as expected.
Could you please try uninstalling your current PP and installing PP from the zip archive? (see Download section).

I tried to uninstall pp as follows:

root@user-Sys:~# easy_install -m pp

And then I deleted the egg at  /usr/local/lib/python2.6/dist-packages/pp-1.6.1-py2.6.egg

I then reinstalled pp as follows:

root@user-Sys:~# easy_install pp

...with the resulting output:
____________________________________________________________
Searching for pp
Reading http://pypi.python.org/simple/pp/
Reading http://www.parallelpython.com
Best match: pp 1.6.1
Downloading http://www.parallelpython.com/downloads/pp/pp-1.6.1.zip
Processing pp-1.6.1.zip
Running pp-1.6.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-WDrDC6/pp-1.6.1/egg-dist-tmp-8z9ZnA
warning: no files found matching 'python-restlib.spec'
zip_safe flag not set; analyzing archive contents...
pp: module references __file__
ppworker: module references __file__
Adding pp 1.6.1 to easy-install.pth file
Installing ppserver.py script to /usr/local/bin

Installed /usr/local/lib/python2.6/dist-packages/pp-1.6.1-py2.6.egg
Processing dependencies for pp
Finished processing dependencies for pp
_______________________________________________________

Notice in the above output the warning:
warning: no files found matching 'python-restlib.spec'
zip_safe flag not set; analyzing archive contents...

I don't know if that is relevant to my problem (probably not).

Any other ideas?  What else I can try to get pp to work correctly on my system?
It really looks like pp is the way to go, but my Ubuntu 10.10 system isn't cooperating.
Thanks for any ideas!    :-)

-Rafael (mathprof)