mutilple spider via fork()
 

Well I read that one message about using the perl wrapper script to seperate spider tasks:
http://www.phpdig.net/showthread.php?s=&threadid=934


Then the one about the php wrapper:
http://www.phpdig.net/showthread.php?threadid=662

What I am thinking of trying is using pcntl_fork() to split the spidering and indexing processes.

Another thing I am going to take a look at is running the phpdig code through Zend Performance Studio and running it on a server with Zend optimizer to see how much more speed I can get out of it.

I really like phpdig and it suits all my needs. If I can get it to run faster all the better.

Anyone have any comments or experience info on trying any of the above mentioned.

No experience with that forking function (bad joke there) but good luck! I have a feeling that leftover gunk in the temporary spider table might confuse multiple spiders, but we'll see I guess.