Title: using pp for job submission of python scripts
Post by: ppgeo on July 03, 2012, 06:16:01 AM
I'm developing a system where I want to use pp for executing external python scripts. These scripts are often numpy calculations that require 0.5sec to 2min and may have other dependencies too.
For each job, the only input is a module name (i.e. "mymodule.py"), function name (i.e. "myfunction"), and a string input to the function (i.e. "/tmp/in.tif").
I've been using the "proxy" approach as described elsewhere in this forum, plus python's __import__(...) and getattr(...). The following code seems to work but I feel that there may be better ways to do this, thus I'm asking for your advice.
import numpy as np
return filename + internalfunction()
modulestr = "mymodule"
funcstr = "myfunction"
argstr = "/tmp/in.tif"
def execfunc(modulestr, funcstr, argstr):
''' proxy function '''
coremodule = __import__(modulestr)
mainfunction = getattr(coremodule, funcstr)
ret = mainfunction(argstr)
job_server = pp.Server()
job1 = job_server.submit(execfunc, args=(modulestr,funcstr,argstr))
r1 = job1()