Parallel Python
Home arrow Forums arrow Python Forums arrow Parallel Python Forum arrow subprocess or pp error?
Parallel Python Community Forums rss  
April 18, 2014, 06:46:57 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Parallel python forum is up and running!
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: subprocess or pp error?  (Read 5618 times)
0 Members and 1 Guest are viewing this topic.
mgag
Jr. Python
**

Karma: 0
Posts: 5


View Profile
« on: January 10, 2010, 02:21:03 PM »

I am learning and testing out pp and got a little stuck with a simple example. 

Code:
import pp
import subprocess

ppservers = ()
job_server = pp.Server(ppservers=ppservers)
job_server.print_stats()

def doJob( cmd ):
    subprocess.call( cmd, shell=True )

job1 = job_server.submit(doJob, args=("./prog",), modules=("subprocess",) )
job2 = job_server.submit(doJob, args=("./prog",), modules=("subprocess",) )
job3 = job_server.submit(doJob, args=("./prog",), modules=("subprocess",) )

job_server.print_stats()
job_server.wait()
job_server.print_stats()

The output is,

Code:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pp.py", line 762, in __run
    sresult = worker.t.receive()
  File "/usr/lib/python2.6/site-packages/pptransport.py", line 133, in receive
    msg = self.r.read(msg_len)
MemoryError
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0x7f513a8fbf10>>
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pp.py", line 771, in __run
    job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pp.py", line 762, in __run
    sresult = worker.t.receive()
  File "/usr/lib/python2.6/site-packages/pptransport.py", line 133, in receive
    msg = self.r.read(msg_len)
MemoryError
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0x7f513a8fbf10>>
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pp.py", line 771, in __run
    job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pp.py", line 762, in __run
    sresult = worker.t.receive()
  File "/usr/lib/python2.6/site-packages/pptransport.py", line 133, in receive
    msg = self.r.read(msg_len)
MemoryError
Unhandled exception in thread started by <bound method Server.__run of <pp.Server object at 0x7f513a8fbf10>>
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pp.py", line 771, in __run
    job.finalize(sresult)
UnboundLocalError: local variable 'sresult' referenced before assignment

It may very well be an error with "prog" - but at this point I don't think it is.
I am running on Fedora Core 11, with a quad core intel.  The program is running on a local machine.  I can tell from the cpu usage that all three jobs are run and take the expected amount of time.  The memory usage seems to be very small...



Logged
Midnighter
Full Python
***

Karma: 0
Posts: 13


View Profile
« Reply #1 on: January 10, 2010, 04:21:28 PM »

I have the same error when trying to use subprocess or any other kind of system call. See also this post. I'd be very much interested in the origin of it.
« Last Edit: January 10, 2010, 04:23:32 PM by Midnighter » Logged
mgag
Jr. Python
**

Karma: 0
Posts: 5


View Profile
« Reply #2 on: January 10, 2010, 05:12:41 PM »

There is another post in here that hints the problem might be stdout, that is, if the subprocess writes to stdout it may interfere with pp.  I will try modifying my program and see if that helps.

Is there a way to tell if all the jobs queued have been processed?
Logged
Midnighter
Full Python
***

Karma: 0
Posts: 13


View Profile
« Reply #3 on: January 11, 2010, 02:43:32 AM »

You are absolutely right! If one doesn't resort to the subprocess convenience functions but instead connects to the command's stdin, stdout, and stderr it works. E.g.:

Code: (python)
p = subprocess.Popen(["ls", "-l"], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
   stderr=subprocess.PIPE, shell=True)
(stdout, stderr) = p.communicate()
print p.returncode
print stdout
 
Logged
mgag
Jr. Python
**

Karma: 0
Posts: 5


View Profile
« Reply #4 on: January 11, 2010, 05:10:19 PM »

Yes that seemed to work for me to.

I am now trying to run the test program as a cluster, but I have only one Unix box.  So in one terminal window I run ppserver.py, and in another window I try and run the job, the code looks like,

Code:
import pp
import os, time, sys
import subprocess

ppservers = ('local',)
job_server = pp.Server(ppservers=ppservers)

def doJob( ):
    print cmd
    p = subprocess.Popen(["./prog", ""], stdin=subprocess.PIPE, stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
    #(stdout, stderr) = p.communicate()

job1 = job_server.submit(doJob, args=(None,), modules=("subprocess",) )
result = job1()
job_server.wait()
job_server.print_stats()

However it fails to execute prog correctly, and I am blind to the error - no output to stdout (which I don't mind, I just want it to work).  Questions and observations?
1) Can I run the ppserver on a local machine and send jobs to it from the local machine?
2) When prog runs, I see TWO progs in the processes list - it seems two are started, which might explain why prog is crashing (prog needs access to temporary files it creates, if it can't create the files it will stop).
3) The code above does not stop on result = job1() or on job_server.wait().  Shouldn't it?
Logged
Midnighter
Full Python
***

Karma: 0
Posts: 13


View Profile
« Reply #5 on: January 12, 2010, 09:05:55 AM »

I tried around some more and I have the following suggestions for you, first of all I had to not use shell=True (default is False anyway) when trying this and I would insert p.wait() just to make sure the command ends before proceeding and so that you get p.returncode even if you're not interested in the stdout.

Code: (python)
p = subprocess.Popen(["./prog", ""], stdin=subprocess.PIPE, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p.wait()
 

Hope that helps.
Logged
mgag
Jr. Python
**

Karma: 0
Posts: 5


View Profile
« Reply #6 on: January 12, 2010, 04:54:18 PM »

I think I see my problem, and/or limitation of ppserver.  The client python application has to be running in order for the job to execute in the server.  I was trying to post jobs and have them complete in the server after the client application (the one that posted the jobs) has terminated.
Logged
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.16 | SMF © 2011, Simple Machines Valid XHTML 1.0! Valid CSS!
Nutrition facts and analysis