still with the same script as my 'sqlite...' thread but new waters. In the past I have run a new instance of the script on the computers available to process. Now I've switched to a cluster method and am getting some unexpected results.

First, when processing only with local cpus output from each task would only print if I included:

Now using a cluster method output will print without the above 'for' loop.

And second (related to the first), the above loop does not work and sadly my script depends on it. Well, it does work but 'job()' fails. on the first iteration of the loop the script perpetually waits at job() even though all the worker threads are chugging along completing images. Oddly enough the callback functions fine and is run every time a worker finishes. atm I have no idea what is causing the hold up and am wondering if any here have any pointers. The pp basics of my scripts are:


jt

After more playing around I've made some progress but am still encountering an odd problem. I have a lot of jobs to process and to make the whole thing more manageable I break the overall job up into smaller batches. job queue is filled and when it's finished processing, is filled again and on and on. Whats occurring now is on the first iteration of the loop: the queue is filled; the 'for image, job in jobs' hold on it's first iteration at job() but the callback function is called at the time any worker completes it's task; all the jobs process normally and when the first iteration of the job queue is complete the 'for image' job in jobs' loop blasts through with all the correct job() output.

On the second iteration of the job queue the 'for image, job in jobs' loop runs correctly and job() outputs it's results as it's received along with the callback function.

Though the second iteration of the job queue works, I can't call it reliable as a database needs to be updated at the time a worker completes.

Any ideas?
jt

Thanks for posting your notes!

By the way, instead of batches you can use callback argument.
The callback function is called when a job completes and this is a perfect time to add a new job to the queue (assuming your jobs are so big that they cannot be submitted).