PDA

View Full Version : Number of ' spider ' simultaneous


fr :: anonymus
12-14-2003, 08:33 AM
Hi,

i would like to know how many spider can we launch simultaneously. So : Is it possible, and no damageous to have two windows like that ..../admin/spider.php?site_id=........ ?

Thanks.

Charter
12-14-2003, 04:31 PM
Hi. Assuming you mean to use the same database tables, I'm not sure that PhpDig will 'mutlithread' appropriately, especially if two spiders hit the same site. If you are looking to crawl somewhat faster, I would suggest crawling from the command line to circumvent the browser interface.

fzxdude
01-24-2004, 10:06 AM
I have about 450 domains spidered with it so far ... run up to about 12 here with no troubles ... things do slow down (which isn't so bad cuz I'm not hammering the other sites so hard)