Parallel Python Community Forums

Quite new to pp.  I was looking to parallelize an executable called from python using subprocess but seemed to have run into an error with ppworker.  I would like to be able to call the executable with command like syntax 'birrp5<scriptfile.script' for as many processors as a computer might have.

the code is :

def callBIRRP(scriptfile,birrploc=r'c:\Peacock\PHD\BIRRP'):
    os.chdir(birrploc)
    #subprocess.call([r"c:\Program Files\Notepad++\notepad++.exe",scriptfile])
    subprocess.os.system('birrp5<'+file)
    print 'Running Script file:'+file
    return os.path.dirname(file)

scriptfiles=(r"D:\Data\S01\S01010\Comb09to19\BF\S01.script",
            r"D:\Data\S01\S01010\Comb09to19\BF\S01.script",)

jobserver=pp.Server()
 
print "Starting pp with", jobserver.get_ncpus(), "workers"
 
jobs = []
for file in scriptfiles:
    jobs.append(jobserver.submit(callBIRRP,args=scriptfiles,
                                 modules=("os","subprocess",)))
 
for job in jobs:
    print job()
 
jobserver.print_stats()

and the error is:

runfile(u'C:\\Peacock\\Python\\MTPro\\src\\pptest.py', args='')
Starting pp with 2 workers
An error has occured during the function execution
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\ppworker.py", line 97, in run
    __result = __f(*__args)
  File "<string>", line 2, in callBIRRP
WindowsError: [Error 267] The directory name is invalid: 'D:\\Data\\S01\\S01010\\Comb09to19\\BF\\S01.script'
None
An error has occured during the function execution
Traceback (most recent call last):
  File "C:\Python26\lib\site-packages\ppworker.py", line 97, in run
    __result = __f(*__args)
  File "<string>", line 2, in callBIRRP
WindowsError: [Error 267] The directory name is invalid: 'D:\\Data\\S01\\S01010\\Comb09to19\\BF\\S01.script'
None
Job execution statistics:
 job count | % of all jobs | job time sum | time per job | job server
         2 |        100.00 |       0.0000 |     0.000000 | local
Time elapsed since server creation 0.0150001049042

So when I run callBIRRP from a python shell it works fine, so the file paths are correct, but something seems to happen within ppworker, any hints?

Cheers

Should be: