PhpDig.net - Multiple Spiders

PhpDig.net (http://www.phpdig.net/forum/index.php)

- How-to Forum (http://www.phpdig.net/forum/forumdisplay.php?f=33)

- - Multiple Spiders (http://www.phpdig.net/forum/showthread.php?t=1646)

jmitchell

12-16-2004 11:57 AM

Multiple Spiders

Hi there,

I have just installed phpdig 2 days and I'm very excited about using it! I have been spidering for almost 2 days, and I would like to know, is there any way to run multiple spiders?

I have seen several threads asking the same question, and some of them had answers, but can I get some tested code that is being used currently for multiple spiders?

thanks

J Mitchell

Charter

12-16-2004 02:15 PM

Maybe you can get an idea from this thread or try using multiple browser windows.

jmitchell

12-16-2004 03:52 PM

so, I can have multiple browsers on multiple pcs running the spider at the same time and it will work?

Charter

12-16-2004 04:43 PM

If you are going to use multiple browsers, then you might want to comment out the following line in the spider.php file:

PHP Code:


		
			
array_unshift($list_sites,$site_datas);

If you are going to use shell, then you might want to send in only one site per process.

Either way, you should test it out on some of your demo sites and see if it works, whether you need to do other modifications to the code, etcetera.

All times are GMT -8. The time now is 05:14 AM.