Parallel Python Community Forums

Python Forums => Parallel Python Forum => Topic started by: Labrat on June 08, 2016, 09:02:48 AM



Title: Parallel Python
Post by: Labrat on June 08, 2016, 09:02:48 AM
Hello all, I just downloaded pp as I figured it would help with running some of my scripts faster. However when trying to implement it with a subsampling script, I am getting an error. Any help with this?

Script:
Code:
import sys, random, itertools
import HTSeq
import pp

#eneter arguments as so (make sure you are in the directory where output files will go): python HTSeq_subsample.py <mate1> <mate2>
def subsample():
fraction = float(.3)

in1 = sys.argv[1]
in2 = sys.argv[2]
iter1 = iter( HTSeq.FastqReader( in1 ) )
iter2 = iter( HTSeq.FastqReader( in2 ) )

output1 = in1 + "_sub_sample_" + str(fraction)
output2 = in2 + "_sub_sample_" + str(fraction)
out1 = open( output1, "w" )
out2 = open( output2, "w" )

for read1, read2 in itertools.izip( iter1, iter2 ):
   if random.random() < fraction:
      read1.write_to_fastq_file( out1 )
      read2.write_to_fastq_file( out2 )
     
out1.close()
out2.close()

ppservers = ()
job_server = pp.Server(ppservers=ppservers)
print("Starting parrallel processing with", job_server.get_ncpus(), "workers")
job1 = job_server.submit(subsample())
result = job1

Error message:
Code:
Traceback (most recent call last):
  File "/Volumes/cachannel/RNA_SEQ/Notch_RNASeq/in_silico_test/HTSeq_subsample_.3.py", line 30, in <module>
    job1 = job_server.submit(subsample())
  File "//anaconda/lib/python2.7/site-packages/pp.py", line 460, in submit
    sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
  File "//anaconda/lib/python2.7/site-packages/pp.py", line 638, in __dumpsfunc
    sources = [self.__get_source(func) for func in funcs]
  File "//anaconda/lib/python2.7/site-packages/pp.py", line 705, in __get_source
    sourcelines = inspect.getsourcelines(func)[0]
  File "//anaconda/lib/python2.7/inspect.py", line 690, in getsourcelines
    lines, lnum = findsource(object)
  File "//anaconda/lib/python2.7/inspect.py", line 526, in findsource
    file = getfile(object)
  File "//anaconda/lib/python2.7/inspect.py", line 420, in getfile
    'function, traceback, frame, or code object'.format(object))
TypeError: None is not a module, class, method, function, traceback, frame, or code object


Title: Re: Parallel Python
Post by: xen on November 26, 2017, 12:14:23 PM
Try
Code:
job1 = job_server.submit(subsample)
instead of
Code:
job1 = job_server.submit(subsample())
which passed None to PP.