View Full Version : Number of ' spider ' simultaneous

fr :: anonymus
12-14-2003, 08:33 AM

i would like to know how many spider can we launch simultaneously. So : Is it possible, and no damageous to have two windows like that ..../admin/spider.php?site_id=........ ?


12-14-2003, 04:31 PM
Hi. Assuming you mean to use the same database tables, I'm not sure that PhpDig will 'mutlithread' appropriately, especially if two spiders hit the same site. If you are looking to crawl somewhat faster, I would suggest crawling from the command line to circumvent the browser interface.

01-24-2004, 10:06 AM
I have about 450 domains spidered with it so far ... run up to about 12 here with no troubles ... things do slow down (which isn't so bad cuz I'm not hammering the other sites so hard)