Hello!

I'm new in parallel python, and I'm trying to use it to distribute the computation of a "big" tool (gdal2tiles, if you've heard of it). I was running a few simple tests to get familiar with pp, but I ran into a problem that I can't explain.

Here is the code :

The "dir" in subprocess.Popen is just here to pass a valid shell command.

On my ppserver, I run "ppserver.py -a -d".

Here is what I get on the client's console :

2013-05-22 16:36:39,128 Listening (0.0.0.0, 60000)
2013-05-22 16:36:39,144 Task 0 submited, function='fun'
2013-05-22 16:36:39,145 Client sends initial broadcast to (255.255.255.255, 60000)
2013-05-22 16:36:39,145 Discovered host (192.168.0.177, 60000) message=C
2013-05-22 16:36:39,148 Discovered host (192.168.0.177, 60000) message=S
2013-05-22 16:36:39,148 Connecting to host 192.168.0.177:60000
2013-05-22 16:36:39,148 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,151 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,153 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,154 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,157 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,158 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,161 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,163 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,164 Creating Rworker id=192.168.0.177:60000 persistent=False
2013-05-22 16:36:39,165 Connected to ppserver (host=192.168.0.177, port=60000)                     with 4 workers
2013-05-22 16:36:39,167 Task (remote) 0 started
2013-05-22 16:36:44,520 Discovered host (192.168.0.177, 60000) message=S
2013-05-22 16:36:54,523 Discovered host (192.168.0.177, 60000) message=S

And on the server :

2013-05-22 16:45:07,234 - pp - INFO - Creating server instance (pp-1.6.4)
2013-05-22 16:45:07,240 - pp - INFO - Running on Python 2.6.2 win32
2013-05-22 16:45:07,571 - pp - INFO - pp local server started with 4 workers
2013-05-22 16:45:07,571 - pp - DEBUG - Strarting network server interface=0.0.0.0 port=60000
2013-05-22 16:45:07,572 - pp - DEBUG - Listening (0.0.0.0, 60000)
2013-05-22 16:45:07,573 - pp - DEBUG - Server sends broadcast to (255.255.255.255, 60000)
2013-05-22 16:45:07,575 - pp - DEBUG - Discovered host (192.168.0.177, 60000) message=S
2013-05-22 16:45:09,927 - pp - DEBUG - Discovered host (192.168.0.177, 60000) message=C
2013-05-22 16:45:09,927 - pp - DEBUG - Replying to host 192.168.0.177:60000
2013-05-22 16:45:09,928 - pp - DEBUG - Discovered host (192.168.0.177, 60000) message=S
2013-05-22 16:45:09,930 - pp - DEBUG - Control message received: STAT
2013-05-22 16:45:09,933 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,934 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,934 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,937 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,938 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,940 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,941 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,944 - pp - DEBUG - Control message received: EXEC
2013-05-22 16:45:09,944 - pp - DEBUG - Task 0 inserted
2013-05-22 16:45:09,946 - pp - INFO - Task 0 started
2013-05-22 16:45:09,969 - pp - DEBUG - Exception in _run_local (possibly expected)
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\pp.py", line 722, in _run_local
    sresult = worker.t.receive()
  File "C:\Python26\lib\site-packages\pptransport.py", line 148, in receive
    msg = self.r.read(e_size-r_size)
OverflowError: long int too large to convert to int
2013-05-22 16:45:09,970 - pp - DEBUG - Task 0 ended
2013-05-22 16:45:17,575 - pp - DEBUG - Server sends broadcast to (255.255.255.255, 60000)
2013-05-22 16:45:17,576 - pp - DEBUG - Discovered host (192.168.0.177, 60000) message=S
2013-05-22 16:45:27,576 - pp - DEBUG - Server sends broadcast to (255.255.255.255, 60000)
2013-05-22 16:45:27,578 - pp - DEBUG - Discovered host (192.168.0.177, 60000) message=S

You see that "Task 0" ends on the server but not on the client : that's what I'd like to understand.

Then, I end the client and here what's appear on the server :

Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 368: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "C:\Python26\lib\logging\__init__.py", line 773, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 479: ordinal not in range(128)
2013-05-22 16:45:31,329 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,332 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,336 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,339 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,342 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,345 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,348 - pp - DEBUG - Closing client socket
2013-05-22 16:45:31,351 - pp - DEBUG - Closing client socket

Before going any further in my tests, I would like to understand what's happening here, because my final program is way more complicated.

Thanks!

Hi there,

I've got this function and I'm trying to parallelize. I think the key here is to properly slice the 2d arrays and to watch out race conditions if any.

The function to parallelize with parallel python is the interpol, which calls a c wrapped function from a library that takes scalar values (interpolate(lat,lon)):


How to partition the job and then properly merge the results?

Many thanks in advance!!

Hi all, what's the best practice for launch
ppserver under supervisord?

I'm finding some problem becuase killing the main thread don't kill all process.

any ideas?

thank you
Glauco

Hi, I am trying to use PP to basically execute a series of scripts which don't return anything in parallel. Everything I have tried seems to just execute in sequence instead on concurrent. Any help or advice on this appreciated. My code is below.

   

The script actually inters across a range of ints. I have tried also this variation as well as placing the actual for code in different scripts -- they all execute but execute in sequence.


I will keep plugging at it -- but just in case anyone might have done this before and has some tips. The examples on the site are great but they all seem to be mathematical which I am not actually using.

Solved.

Hi,

I am having troubles in running PP code after rewriting my code to use objects (instead plain functions as it was before; it worked).

The problem is that I am not sure how to submit and initialize an object and submit it to PP; the class has __init__ which defines the global properties of the object;

The error i get is (for each of my 15 different processes to run):

An error has occured during the module import
Traceback (most recent call last):
  File "/home/jseva/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/ppworker.py", line 49, in preprocess
    exec module
  File "<string>", line 1, in <module>
  File "/home/jseva/SemanticVIRT/classification(H1)/classification_createModels.py", line 174, in <module>
    ncpus = int(sys.argv[1])
ValueError: invalid literal for int() with base 10: '2>/dev/null'
An error has occured during the function execution
Traceback (most recent call last):
  File "/home/jseva/epd_free-7.3-2-rh5-x86_64/lib/python2.7/site-packages/ppworker.py", line 90, in run
    __result = __f(*__args)
  File "<string>", line 17, in __init__
SystemExit: Wrong 'type' parameter in createData.__init__

I start PP with:

ppservers = ()
#ppservers = ("10.0.0.1",)

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)

print "Starting pp with", job_server.get_ncpus(), "workers"
start_time = time.time()

#initialize and start
rootDirectory = raw_input("Input root directory to store files (keyboard input): ")
"""
print dataModel.GROUPTYPE
print dataModel.percentageList
print dataModel.rootDir
"""
#start PP
jobs = []
inputs = ShevaDB().getMainCat()
for index in inputs:
    print index
    dataModel = createData(2,rootDirectory)
    print dataModel.GROUPTYPE
    print dataModel.percentageList
    print dataModel.rootDir
    jobs.append(job_server.submit(dataModel.createData,
                                  (index,),
                                  depfuncs = (),
                                  modules = ("time","classification_createModels","ShevaDB","ShevaTPF","ShevaUtils","ShevaVect",)))

for job in jobs:
    result = job()
    if result:
        break
#prints
job_server.print_stats()
print "Time elapsed: ", time.time() - start_time, "s"

My class __init__ code is:
class createData:
   
    def __init__(self,type,rootDir):
        """
        type: 1 -> full data
              2 -> limited data
        """

        #percentage of data to be used for model build
        if type == 1:
            self.GROUPTYPE = ["CATID","FATHERID","GENERAL"]
            self.percentageList = [10, 25, 50, 75, 100]
        elif type == 2:
            self.GROUPTYPE = ["GENERAL"]
            self.percentageList = [10]
        else:
            sys.exit("Wrong 'type' parameter in createData.__init__")
       
        self.shevaDB = ShevaDB()
        self.shevaTPF = ShevaTPF()
        self.shevaUtils = ShevaUtils()
        self.shevaVect = ShevaVect()
       
        if rootDir != "":
            self.shevaUtils.createDirOne(str(rootDir))
            self.rootDir = str(rootDir)
        else:
            sys.exit("Wrong 'rootDir' parameter in createData.__init__")
   
    def createData(self, category):
       ... code of createData function

PP initialization and class are in the same py file.

Any ideas?

Best,
J.

Hi all,
I've solved my problem, this is a follow-up for those who may have had similar problems.  The cause of the slow-down was the networked cluster file-system.  Each of my executables continuously updated 4 monitoring files (with small amounts of data), 48 * 4 open files on the network drive - this must have caused network issues.  Writing these files to the local disk instead, removed the slow-down.

One thing I can't explain: there was no problem when Python was not used.  The problem arose only when each executable was started from within its own Python interpreter (the parallelism model that both PP and the multiprocessing module use) - it was therefore not specific to PP.

The only explanations I can think of are (a) that Python *somehow* changed the executable's file-IO buffering behaviour, or (b) each interpreter itself requires multiple open files, pushing the number of open files too high.

Hope that helps someone...
Richard

i am trying to run the code on my computer and sending the tasks on a virtual machine  using the auto detect but it does not seem to detect the node even when i do  node-1> ./ppserver.py -a  on the nodes (and i am not getting any error message)..has anyone of u worked with virtual machines? maybe you could give me an idea why it is not working? please i am really new to all this thanks..

Hi all,
I'm having performance problems with PP.  I'm trying to run 48 separate instances of a sequential CFD code on a single node with 48 cores.  Since everything is trivially parallelizable I expect roughly linear speedup.

My code is an executable, which I'm starting with subprocess.call().  If start the code with a simple for-loop of calls(), backgrounding each one; then all 48 instances run at approximately the same speed as expected.  If instead use PP and submit() a function with a call() inside, each instance runs at a different speed, the slowest being ~100 times slower than expected.

Memory is not an issue, there's not much I/O going on (just adding lines to the end of some monitoring files), I've tried changing stdout buffering to no effect.

Does anyone have any idea what might be happening here?  Something to do with threads maybe?
Best regards,
Richard

P.s. Is there a better way of getting error information back from the jobs, other than the "write to file" approach used below?
 

Hello,

I tried the example sum_primes.py, running it from the bash shell works fine. However, when I submit this code within a python instance, I get a bunch of errors. Does pp work at all within python or is it good just for scripts called from a shell, such as bash?


I took care of removing the section

and use instead

which defaults to 4.

I installed the pp module with "pip install pp".

using python 2.7.3

Thank you.