Hi,

i'm using pp to run a huge number of simulations (a genetic algorithm) and in the middle of the simulation after a couple of days it either hungs the machine or breaks with the message:

Traceback (most recent call last):
  File "iorga.py", line 75, in <module>
  File "/home/jorge/dev/schoner/dftgenalg/ga_mod.py", line 506, in evolve
  File "iorga.py", line 17, in applyFitnessFunction
  File "/home/jorge/dev/schoner/dftgenalg/dftparallel.py", line 68, in run
  File "/home/jorge/dev/schoner/dftgenalg/dftparallel.py", line 24, in __init__
  File "/var/lib/python-support/python2.5/pp.py", line 282, in __init__
  File "/var/lib/python-support/python2.5/pp.py", line 437, in set_ncpus
  File "/var/lib/python-support/python2.5/pp.py", line 121, in __init__
  File "/usr/lib/python2.5/popen2.py", line 189, in popen3
  File "/usr/lib/python2.5/popen2.py", line 52, in __init__
OSError: [Errno 24] Too many open files
Exception exceptions.AttributeError: AttributeError("Popen3 instance has no attribute 'pid'",) in <bound method Popen3.__del__ of <popen2.Popen3 instance at 0x2bf8dd0>> ignored

Finally, i discovered that the problem is that destroying the server (i'm doing this because of a problem with ctypes that otherwise hangs the program with a misterious double free) the pipes aren't destroyed.
In another thread i read a suggestion of adding a self.r.close() self.w.close() in the destroy method, which Vitaly said would consider for inclusion.
In the code of pp.py it exits the function PPTransport.close(), which does close both pipes, but the destroy method doesn't call this, so pipes exists forever. I found that changing this method solves my problem and I considered on sharing this. Is there a better way to accomplish this? The new code is:

 def destroy(self):
        """Kills ppworkers and closes open files"""
        self.__exiting = True
        self.__queue_lock.acquire()
        self.__queue = []
        self.__queue_lock.release()

        for worker in self.__workers:
            worker.t.exiting = True
            if sys.platform.startswith("win"):
                os.popen('TASKKILL /PID '+str(worker.pid)+' /F')
            else:
                try:
                    worker.t.close() # This is the new line
                    os.kill(worker.pid, 9)
                    os.waitpid(worker.pid, 0)
                except:
                    pass

Thanks a lot for this nice program.

(fixed in PP 1.6 RC4 or later)

I am using 1.6.1, and still get IOError: [Errno 24] Too many open files. After destroy job_server, resource cannot be released. The number of open files keep increasing. Only remote server has this issue.

I have the same problem with pp 1.6.1, the number of sockets is increasing until it hits the too-many-files limit. When I run this code with one client with 2 workers it stops after approx. 140 itterations. Then ls -l /proc/PID/fd shows > 1000 files

I understand better what's happening. My jobs are long lived, and the hardcoded 10 sec timeout setting in pptransport.py is too small.

As a result, the connection is dropped after 10 sec, the job still (!) runs to completion thus blocking the new incoming requests, the attempt to write back the result ends in an exception, and the socket sits in CLOSE_WAIT. Incoming requests keep piling up, waiting and timing out, too, adding to the sockets in CLOSE_WAIT.

Granted, there is a FAQ entry suggesting to change the variable. It is a workaround, but I got the pp module installed via the standard Ubuntu way, and hacking the source in /usr/share/sharedpy just does not feel right.

I suggest this setting becomes a programmable parameter. The behavior when the timeout occurs and the connection is dropped also does not seem very graceful to me.

Great code though. Very simple to set up and use.