Parallel Python

I'm developing a system where I want to use pp for executing external python scripts. These scripts are often numpy calculations that require 0.5sec to 2min and may have other dependencies too.

For each job, the only input is a module name (i.e. "mymodule.py"), function name (i.e. "myfunction"), and a string input to the function (i.e. "/tmp/in.tif").

I've been using the "proxy" approach as described elsewhere in this forum, plus python's __import__(...) and getattr(...). The following code seems to work but I feel that there may be better ways to do this, thus I'm asking for your advice.

#mymodule.py

Code:

#mymodule.py
import time
import numpy as np

def myfunction(filename):
return filename + internalfunction()

def internalfunction():
return str(np.arange(10))

#pp_system.py

Code:

#pp_system.py
import pp

modulestr = "mymodule"
funcstr = "myfunction"
argstr = "/tmp/in.tif"

def execfunc(modulestr, funcstr, argstr):
''' proxy function '''
coremodule = __import__(modulestr)
mainfunction = getattr(coremodule, funcstr)
ret = mainfunction(argstr)
return ret

job_server = pp.Server()

job1 = job_server.submit(execfunc, args=(modulestr,funcstr,argstr))

r1 = job1()
print r1

	Author	Topic: using pp for job submission of python scripts (Read 786 times)
0 Members and 1 Guest are viewing this topic.