PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   How-to Forum (http://www.phpdig.net/forum/forumdisplay.php?f=33)
-   -   Number of ' spider ' simultaneous (http://www.phpdig.net/forum/showthread.php?t=292)

fr :: anonymus 12-14-2003 08:33 AM

Number of ' spider ' simultaneous
 
Hi,

i would like to know how many spider can we launch simultaneously. So : Is it possible, and no damageous to have two windows like that ..../admin/spider.php?site_id=........ ?

Thanks.

Charter 12-14-2003 04:31 PM

Hi. Assuming you mean to use the same database tables, I'm not sure that PhpDig will 'mutlithread' appropriately, especially if two spiders hit the same site. If you are looking to crawl somewhat faster, I would suggest crawling from the command line to circumvent the browser interface.

fzxdude 01-24-2004 10:06 AM

I have about 450 domains spidered with it so far ... run up to about 12 here with no troubles ... things do slow down (which isn't so bad cuz I'm not hammering the other sites so hard)


All times are GMT -8. The time now is 01:47 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.