View Full Version : mutilple spider via fork()

05-17-2004, 10:46 PM
Well I read that one message about using the perl wrapper script to seperate spider tasks:

Then the one about the php wrapper:

What I am thinking of trying is using pcntl_fork() to split the spidering and indexing processes.

Another thing I am going to take a look at is running the phpdig code through Zend Performance Studio and running it on a server with Zend optimizer to see how much more speed I can get out of it.

I really like phpdig and it suits all my needs. If I can get it to run faster all the better.

Anyone have any comments or experience info on trying any of the above mentioned.

05-18-2004, 02:21 AM
No experience with that forking function (bad joke there) but good luck! I have a feeling that leftover gunk in the temporary spider table might confuse multiple spiders, but we'll see I guess.