Number of ' spider ' simultaneous

fr :: anonymus · 12-14-2003, 08:33 AM

Hi,

i would like to know how many spider can we launch simultaneously. So : Is it possible, and no damageous to have two windows like that ..../admin/spider.php?site_id=........ ?

Thanks.

Charter · 12-14-2003, 04:31 PM

Hi. Assuming you mean to use the same database tables, I'm not sure that PhpDig will 'mutlithread' appropriately, especially if two spiders hit the same site. If you are looking to crawl somewhat faster, I would suggest crawling from the command line to circumvent the browser interface.

fzxdude · 01-24-2004, 10:06 AM

I have about 450 domains spidered with it so far ... run up to about 12 here with no troubles ... things do slow down (which isn't so bad cuz I'm not hammering the other sites so hard)

12-14-2003, 08:33 AM	#1
fr :: anonymus Green Mole Join Date: Dec 2003 Location: Lyon, France Posts: 17	Number of ' spider ' simultaneous Hi, i would like to know how many spider can we launch simultaneously. So : Is it possible, and no damageous to have two windows like that ..../admin/spider.php?site_id=........ ? Thanks.

12-14-2003, 04:31 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Assuming you mean to use the same database tables, I'm not sure that PhpDig will 'mutlithread' appropriately, especially if two spiders hit the same site. If you are looking to crawl somewhat faster, I would suggest crawling from the command line to circumvent the browser interface. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Number of pages indexed	claudiomet	How-to Forum	0	08-30-2004 02:26 PM
Set time limit on spider.php or number of pages	paullind	Troubleshooting	1	05-01-2004 07:25 AM
pages number limited indexing	JÿGius³	Mod Requests	4	01-13-2004 11:17 AM
Number of links found	bloodjelly	How-to Forum	2	01-09-2004 09:50 AM
Reduce number of connections	druesome	Troubleshooting	1	10-14-2003 07:42 AM

01-24-2004, 10:06 AM	#3
fzxdude Green Mole Join Date: Sep 2003 Posts: 4	I have about 450 domains spidered with it so far ... run up to about 12 here with no troubles ... things do slow down (which isn't so bad cuz I'm not hammering the other sites so hard)