I'm pretty sure the problem is putting the pp.Server call inside the loop.  This is unnecessary as you really only need to create the jobserver once.  Try moving the loop to only cover the actual submissions.  I did that (with a for loop since, well, I always use one) and it worked for me!  Now I *really* know the sums of primes below 100000!

Now, if you are needing to add or remove workers during your program, then you'll want to use autodiscovery.

Matt

Aww...poor Fortran, always forgotten.   

Well, probably, but it's dependent on the cluster.  I use PP on a PBS cluster and I start and stop the servers inside my PBS submission script.  I imagine the same could be done for the other queuing software.

In my case, I use both cores on each node, so I have to remove the "doubling" from $PBS_NODEFILE:


As you can see, I use -a to select auto-discovery, so I also have to use -p in case I don't want to step on other PP jobs running.  I take care of killing off the servers with the timeout, -t.  This way after 40 seconds of no calculation, the servers die and PBS can exit out.

Hmm...I haven't looked at your code too closely, yet, but I can give some advice on doing stuff like this with PP.  In my work, I use PP and subprocess to spawn off jobs like this. 

First, looking at the Popen commands, I think you might be missing something crucial.  I always have to do this:  Ignoring all the split bits, I've found that with Popen, you usually need shell=True, and you'll nearly always need the .communicate() attached to the Popen command or later on to use the results.  I usually refer to Doug Hellman's subprocess intro when I try to use it.

Second, I create and delete temporary directories to take care of the directory junk.  My code has some common files that are generated every time it's run, so it can only be run once in a directory (at the same time, that is).  The code I use to make them is:
  It's fairly easy to figure out, which means I didn't create it (thanks Google search!).  So I do things like: which makes a temporary directory, chdir's into it and I then copy a file (which has an absolute path included in der_input_file) into it.

Hmm...I'm sure I'm forgetting something.  But at the least, try things out with .communicate().  I think that might be the main sticking point.

Actually, the only thing you are doing wrong is not viewing ncpus correctly.  ncpus refers to the number of local workers no matter what, remote nodes are considered only when ncpus=0.  The best way to do what you want to do is to instead consider everyone a remote node, even the local machine.  So, the simple answer is to also include your local machine in the ppservers line: or whatever your local machine IP is.  Note, you'll have to manually start ppserver.py on the local machine with the correct number of workers (the -w switch) just as you did on the remote.  Then, when ncpus=0, the program will use however many ppserver.py processes are running on ppservers.

You can also try autodiscovery as well, which is what I use.  First, on every machine where you want to do PP work, start up ppserver.py with the correct number of workers as usual, and with autodiscovery (-a).  So on the localhost (which you say is dual core), run "ppserver.py -a -w 2".  On the remote computer, if it only has one core, do "ppserver.py -a -w 1".  Now you have ppservers waiting for work.  In your code, you now issue: which tells the code to autodiscover workers.  If ncpus=0, it will find how ever many ppserver.py workers you started.  If ncpus != 0, then it will do local computation only.

Verified as well on Fedora 8, python-2.5.1-15.fc8.x86_64, and pp 1.5.3.  It's like a FORTRAN 77 v. Fortran 2003 bug...stupid first column...

When you create your server using pp.Server, set ncpus=0, e.g.:
This prevents any localhost CPUs from being used, I do believe.

(Note, though, if you are using autodiscovery on, say, a cluster, if you have any ppserver.py instances running on the master, they will be used even with ncpus=0.  This is how I run my jobs under PBS and it allows me to control whether the head node of my job is allowed to do pp work or not.)

Vitalii, I'm at the point where I think I can try an upgrade of PP to 1.5.3 (things have been working with 1.5.2, so I'm a bit skittish).

My main question is how exactly does the pp_secret functionality work on a cluster.  I understand I can add pp_secret="jujube" or whatever to my .pythonrc.py file.  But, do I then need to start my ppserver.py's on the other nodes with a -s switch?  Or do I just need to add the secretword to my Server call in my program?

I believe with autodiscovery this is possible.  (Vitalii will know for sure.)  Here's my reasoning:

When I started using PBS and PP, I was doing some benchmarking using ppserver.py -a and found odd results.  What happened was I submitted, say, a 5, 4, 3, 2, and 1 node job all at the same time.  What I didn't do was assign a unique port number to each job.  So, when I went to look at the results, the timings were all over the place.  The reason was I had 15 servers all looking for work.  So, it didn't matter to PP that PBS had only asked for, say, 1 node, PP saw 15 servers and used them all.

So, if you are allowed on your cluster to have ppserver.py running at all times (like, say, mpd on an MPI system), then if you start them with the -a switch, it should allow for a node to go down and re-upped with no problem--other than the loss of a node.

Vitalii, I am getting to the point of writing up a letter/article (probably letter) about my GA work with PP.  That led to BibTeX work and the question:  how exactly should I cite your work here?  I assume a @Misc entry for the website, but is there corresponding @Article's for the work you've done?  And, do you have any citations of other published work done with PP as well.  Anything to make the references look impressive.

And #2 did.  Before optparse I did #1, but since I'm passing the port number from optparse to PP, I can't do that anymore.

Sigh...I just had to add ten characters...  Stupid Python and its global/local...mutter mutter...

OK...I need some help with what I think is an amazingly simple problem. Essentially, I'm trying to recast a script of mine to use optparse.  Now, to do this, I've been following examples for optparse on the web.  These usually use the:construct.  So, I took a script that did work and recast it into this first.  In doing so, I took my usual statements and put them inside main().  But that seemed to mess with the script as I started getting errors like:
Now, I'm pretty sure my optparse bits are working since I made a separate file and tested them out.  So, all the variables like the port number now occur in the same variables and I don't think they're the problem.  My only guess is that it's the starting of PP inside of main().  And that leads into the whole global/local stuff I've never quite got my hand on in python.

Help.

Just a note in case Mr Milner isn't very active on the board, but it looks like pp got pushed to Fedora 8 Stable yesterday, and it appeared in the massive update that is occurring today.  At least I think I saw it in the insanely huge number of posts to fedora-package-announce.

Oh, and a thank you to Mr Milner.

Huh, you know, writing this out has made me think.  Maybe I'm parallelizing the wrong thing.  Perhaps parallelizing the GA the way I have it now is wrong.  Rather, I could do the GA serially, but just parallelize the DFT calculations.  In the end, there will be much more of those than the GA, and that could scale better.  But, I think if I do that, I might run into load balancing issues at some point.  Eventually, I'd have all large molecules in the queues and sooner or later a worker will be waiting around.

I think that is why I'm thinking of two sets of workers.  A small number for the GA and a large number of DFT calculators.  That way the multiple sets of alphas could keep feeding the DFT workers.  Perhaps in a naive effort to load balance I could random.shuffle() my list of molecules.  That way there would be a mix of diatomics and bigguns all at the same time.  Hmm...I am just not a parallel thinker yet.

Vitalii, what do you think?

Hmm...not quite.  OK, here we go with a bit more specificity.

I have 20 sets, alphas_i, which are like: {'h':0.7,'li':0.67,'be':0.68...}, a set of alphas for elements which are variable data for analytic DFT.  What I then do is for each set, alphas_i, use that information to construct DFT input files for molecules, ['hcl','isobutene','alcl3',...] which, for now, is the extended G2 set (=exG2) of molecules.  So, essentially, each set alphas_i leads to 148 molecule input files as well as 20 or so atomic input files.  DFT is then run on this set of 160 molecules.  After completion, the calculated atomization energy of each of the exG2 molecules is computed and then compared to the experimental value.  I then calculate the mean atomization error of my calculated exG2 set to the benchmark, and use that as the fitness of the set alphas_i.  After completing 20 different sets of 160 DFT calculations, I end up with a list fitness_i = [10,20,60,...] where fitness_j corresponds to the fitness of set alphas_j.  This is then used in the genetic algorithm to calculate a new set of alphas_i for the next generation.  Wash, rinse, repeat.

So, as it is now, I have pp set up so that the submitted job is a set of alphas.  Within that job, the worker does the 160 DFT calcs and returns the fitness (=mean absolute atomization error).  This setup is hamstrung by the fact that it only scales to len[alphas].  What I'm hoping to do at some point is to take this one-D problem into the second dimension by then farming out the 160 jobs each worker does to another set of workers.

For my own sanity, I still think I'll keep around the pp workers that run and gather the DFT results and return the fitness.  My thought was to create two sets of workers, say, 5 that run the overall alphas on one port, and then 10, 20, 30... however many I can, to then handle the DFT calculations underneath on another port.  That is:

fitness_job_server_port1.submit(alphas, exG2set):
   dft_job_server_port2.submit(alphas, molecule):
      return optimized_energy
   return mean_error

What I don't know is if this is the best way to go, or even if pp would like it this way