Okay, folks, this is probably going to be a long, rambling discussio-question mainly for Vitalii since he is the master.  But any other advice is nice.

My question involves my specific problem and using it with PP on a cluster.  First, my problem.  I am using a genetic algorithm in order to optimize some variables used in molecular calculations with the end result to minimize the error of these calculations to benchmark values.  It turns out this problem is very parallel.

1) The genetic algorithm is parallel in that for a population of, say, 20 individuals, each individual may be calculated separately.  In my case, each individual is a set of variables, one per element, that gives different results when run.

2) To calculate the fitness of each individual, I run say ~100 molecular calculations using each set of variables.  Each of these could be done separately.

3) The molecules themselves vary in size and the code that does the calculation can be spread over nodes with MPI.

Right now, my code does #1.  I essentially spawn off each individual as a submitted job to my PP workers.  This has the advantage of being easy to code and simple to picture.  Unfortunately, though, that means I run 100 molecules in serial on each node.  I was wondering if anyone could figure out a way to extend this to #2.  That is, now that I have, say, 1 node for each individual, use a new set of calculation servers to run the 100 molecules in parallel over a few nodes.

As you can see, I don't quite yet understand how to even formulate what I'm imagining, but I'm trying to get to the point where I could, even with 10 sets of variables, efficiently use 100 nodes.  The way it is now, I could 10 efficiently, but any more is a waste.

Thanks for any advice...

Hmm...not quite.  OK, here we go with a bit more specificity.

I have 20 sets, alphas_i, which are like: {'h':0.7,'li':0.67,'be':0.68...}, a set of alphas for elements which are variable data for analytic DFT.  What I then do is for each set, alphas_i, use that information to construct DFT input files for molecules, ['hcl','isobutene','alcl3',...] which, for now, is the extended G2 set (=exG2) of molecules.  So, essentially, each set alphas_i leads to 148 molecule input files as well as 20 or so atomic input files.  DFT is then run on this set of 160 molecules.  After completion, the calculated atomization energy of each of the exG2 molecules is computed and then compared to the experimental value.  I then calculate the mean atomization error of my calculated exG2 set to the benchmark, and use that as the fitness of the set alphas_i.  After completing 20 different sets of 160 DFT calculations, I end up with a list fitness_i = [10,20,60,...] where fitness_j corresponds to the fitness of set alphas_j.  This is then used in the genetic algorithm to calculate a new set of alphas_i for the next generation.  Wash, rinse, repeat.

So, as it is now, I have pp set up so that the submitted job is a set of alphas.  Within that job, the worker does the 160 DFT calcs and returns the fitness (=mean absolute atomization error).  This setup is hamstrung by the fact that it only scales to len[alphas].  What I'm hoping to do at some point is to take this one-D problem into the second dimension by then farming out the 160 jobs each worker does to another set of workers.

For my own sanity, I still think I'll keep around the pp workers that run and gather the DFT results and return the fitness.  My thought was to create two sets of workers, say, 5 that run the overall alphas on one port, and then 10, 20, 30... however many I can, to then handle the DFT calculations underneath on another port.  That is:

fitness_job_server_port1.submit(alphas, exG2set):
   dft_job_server_port2.submit(alphas, molecule):
      return optimized_energy
   return mean_error

What I don't know is if this is the best way to go, or even if pp would like it this way

In this case DFT is the most computationally expensive step and everything else is probably very cheap.
Then the best way would be to start 20 (for each set) x 160 (for each molecule) jobs:
And you should not worry about load balancing because if #jobs >> # cpus PP handles it pretty well.
Then you just collect all the data from 20 x 160 jobs and ether compute fitness serially (if it is very very cheap) or start another 20 PP jobs to compute it.
Does that seem reasonable?