Print Page - using pp for job submission of python scripts

I'm developing a system where I want to use pp for executing external python scripts. These scripts are often numpy calculations that require 0.5sec to 2min and may have other dependencies too.

For each job, the only input is a module name (i.e. "mymodule.py"), function name (i.e. "myfunction"), and a string input to the function (i.e. "/tmp/in.tif").

I've been using the "proxy" approach as described elsewhere in this forum, plus python's __import__(...) and getattr(...). The following code seems to work but I feel that there may be better ways to do this, thus I'm asking for your advice.

#mymodule.py

Code:

#mymodule.py
import time
import numpy as np

def myfunction(filename):
return filename + internalfunction()

def internalfunction():
return str(np.arange(10))

#pp_system.py

Code:

#pp_system.py
import pp

modulestr = "mymodule"
funcstr = "myfunction"
argstr = "/tmp/in.tif"

def execfunc(modulestr, funcstr, argstr):
''' proxy function '''
coremodule = __import__(modulestr)
mainfunction = getattr(coremodule, funcstr)
ret = mainfunction(argstr)
return ret

job_server = pp.Server()

job1 = job_server.submit(execfunc, args=(modulestr,funcstr,argstr))

r1 = job1()
print r1

Parallel Python Community Forums

Python Forums => Parallel Python Forum => Topic started by: ppgeo on July 03, 2012, 06:16:01 AM