When a very lot of links...

[fr :: anonymus]

Hi,

When there are a very lot of links (most than 3000) in the same page, spider record all, and scan them.
But it's very long between 2 links :

17:http://www.*****.net/docs/php/annotee/function.aggregate-properties.php
(temps : 00:49:41)

18:http://www..*****.net/docs/php/annotee/function.aggregate-properties-by-list.php
(temps : 00:54:05)

19:http://www..*****.net/docs/php/annotee/function.aggregate-properties-by-regexp.php
(temps : 00:57:36)


and time, in (at) the end, is very important. (3-5 days ?)

Do you know how to make it faster, without mode localhost ?

Thanks, Anonymus.

[fr :: anonymus]

**** = nexen.

[Charter]

Hi. Are you indexing http://www.nexen.net/docs/php/annotee/indexes.functions.php at level one? If not already done, perhaps try to be selective when indexing a site by choosing the most relevant page to index at level one. Also, if available, perhaps try to index from shell rather than with the browser interface. If indexing from shell, the following can be set in the config file to limit the level to one.

define('SPIDER_MAX_LIMIT',1); //max recurse levels in sipder
define('SPIDER_DEFAULT_LIMIT',1); //default value
define('RESPIDER_LIMIT',1); //recurse limit for update

[fr :: anonymus]

That's for ..../php/annotee/indexes.functions.php

All are in level 1.

Ok, thanks.

Anonymus.